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FOREWORD 


Technological  advances  have  led  to  increased  speed,  mobility,  and 
destructive  power  of  military  operations . So  that  commanders  can  make 
tactical  decisions  consistent  with  the  rapid  change  and  succession  of 
events,  information  on  military  operations  must  be  processed  and  used 
more  effectively  than  ever  before.  To  meet  this  need,  the  Army  is  de- 
veloping automated  systems  for  receipt,  processing,  storage,  retrieval, 
and  display  of  vast  amounts  and  different  types  of  military  data.  There 
is  a concomitant  requirement  for  research  to  determine  how  human  abili- 
ties can  be  used  to  insure  that  command  information  processing  systems 
function  with  maximum  effectiveness. 

Manned  systems  research  in  this  area  done  by  the  U.S.  Army  Research 
Institute  (ARI)  is  directed  toward  enhancement  of  human  performance  and 
improvement  of  man-machine  interaction  in  relation  to  total  system  ef- 
fectiveness. The  research  involves  experimentation  with  various  con- 
figurations of  system  components,  considering  interactions  and  tradeoffs. 
The  end  products — immediate  or  ultimate — are  scientific  findings  on 
human  capabilities  under  varying  conditions  within  the  system.  These 
findings  have  implications  for  system  design,  development,  and  opera- 
tional use.  This  publication  describes  the  evaluation  of  the  relative 
ease  of  learning  two  message  reference  code  sets  used  in  inputting, 
editing,  and  obtaining  displays  of  information  within  the  context  of 
a computer-assisted  tactical  operations  system. 

The  entire  research  effort  is  responsive  to  requirements  of  Army 
Project  2Q062106A723 , "Enhancement  of  Performance  in  Military  Organiza- 
tions," Fiscal  YeaiTX974  Work  Program,  and  to  special  requirements  of 
the  Assistant  Chief  of  Staff  for  Force  Development,  the  Combined  Arms 
Center  of  the  U.S.  Army  Training  and  Doctrine  Command,  and  the  U.S. 

Army  Tactical  Data  Systems  Project. 
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BRIEF 


Requirement : 

To  evaluate  the  relative  ease  or  difficulty  of  learning  two  alter- 
native message  reference  codes  used  during  inserting,  editing,  and  ex- 
tracting information  from  an  Army  tactical  system. 

Procedure : 

Forty  officers  representative  of  potential  G2  and  G3  action  officers 
and  20  enlisted  personnel  representative  of  potential  TOS  input-output 
device  operators  were  given  2 minutes  to  examine  tabular  information 
on  describing  the  message  titles  and  action  in  association  with  a message 
reference  code  consisting  of  either  two  letters  and  a number  (LL#)  or 
four  letters  (LLLL) . Then,  seated  at  a computer  terminal  with  an  input 
keyboard  and  a cathode  ray  tube  (CRT)  display  device,  the  soldiers  in- 
dividually typed  and  transmitted  message  reference  codes  in  response  to 
displays  of  message  titles  and  action  names.  They  had  to  learn  a list 
of  G2  messages  or  G3  messages.  On  the  first  pass  through  the  list, 
titles  were  displayed  in  an  orderly  sequence.  On  subsequent  passes, 
titles  were  in  random  order.  Subjects  had  to  make  an  error-free  pass 
before  they  were  considered  to  have  met  the  criterion  of  learning  and 
were  considered  finished. 


Findings : 

The  error  rate  associated  with  learning  the  LLLL  codes  was  less 
than  half  that  associated  with  learning  the  LL#  codes  (13%  versus  29% 
errors  for  enlisted  personnel  working  on  the  G2  list;  11%  versus  21% 
for  officers  working  on  the  G2  list;  and  7%  versus  15%  for  officers 
working  on  the  G3  list) . Time  required  to  meet  the  learning  criterion 
for  those  learning  the  LLLL  codes  was  about  60%  of  that  required  to 
learn  the  LL#  codes.  Analysis  of  errors  by  character  position  revealed 
that  the  old  code's  numeric  character  and  arbitrary  second  letter  were 
important  sources  of  error. 


Utilization  of  Findings: 

A version  of  the  new  LLLL  code  has  now  been  specified/incorporated 
into  the  Tactical  Operations  System,  Operable  Segment  (TOS2)  computer 
system  as  the  result  of  having  been  jointly  recommended  for  such  incor- 
poration by  members  of  a TOS2  working  group  that  included  the  senior 
author  as  a participant.  The  findings  support  the  previous  hypothesis 
that  the  LLLL  code  would  be  easier  to  learn  than  the  LL#  code  used  by 
Development  Tactical  Operating  System  (DEVTOS)  personnel. 
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INTRODUCTION 

Designers,  developers,  analysts,  and  researchers  of  Army  tactical 
data  systems  continue  to  seek  software,  display,  and  procedural  means 
of  facilitating  the  task  of  translating  free-text  message  information 
into  computer-acceptable  input.  This  report  is  closely  related  to  pre- 
vious Army  Research  Institute  (ARI)  publications''-'2  and  is  part  of  a 
long-term  concern  with  problems  of  information  presentation,  processing, 
and  utilization  in  Command  and  Control  Systems. 

A specific  requirement  for  the  research  reported  here  was  first 
stated  in  early  1972.  Anticipating  the  replacement  of  the  Development 
Tactical  Operating  System  (DEVTOS)  by  the  TOS-operable  Segment  (TOS2) , 
the  Combat  Developments  Command  (CDC)  directed  the  formation  of  an  ad 
hoc  committee  to  (a)  recommend  revisions  of  TOS2  formats,  incorporating 
DEVTOS  and  (MASSTER)  experience  where  applicable;  and  (b)  evaluate  such 
revisions  using  subjective  analysis  and  DEVTOS  testing. 

Representatives  of  CDC,  Modern  Army  Selected  System  Test,  Evaluation, 
and  Review  (MASSTER) , Computer  Systems  Command  (CSC) , Project  Manager, 

Army  Tactical  Data  Systems  (PM  ARTADS) , and  ARI  (then  BESRL)  formed  the 
committee.  Their  work  produced  four  recommendations: 

1.  Consolidation  of  message  formats, 

2.  Use  of  variable  field  data  entry, 

3.  Map  reference  coordinate  "packing,"  and 

4 . Use  of  a new  message  reference  code  system  . 

A very  limited  try-out  of  all  these  revisions  on  the  DEVTOS  facility 
produced  encouraging  results.  However,  it  was  clear  that  more  testing 
was  needed  to  assess  separately  the  influence  of  each  change  on  TOS 
operators’  performance,  to  determine  if  each  change  facilitated  the 
interaction  of  the  operator  with  the  semiautomated  information  system. 


Baker,  J.  D. , Mace,  D.  J.,  and  McKendry,  J.  M.  The  Transform  Operation 
in  TOS:  Assessment  of  the  Human  Component.  ARI  Technical  Research 
Note  212,  August  1969. 
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Strub,  M.  H.  Evaluation  of  Man-computer  Input  Techniques  for  Military 
Information  Systems.  ARI  Technical  Research  Note  226,  May  1971.  (NTIS 
No.  AD  730  315) 
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The  recommended  message  reference  code  set  uses  four  alphabetic 
characters  for  each  member  of  the  code  set.  In  most  cases,  these  char- 
acters comprise  "acronyms"  of  the  message  titles  and  action  names.  The 
DEVTOS  code  set,  by  contrast,  uses  two  alphabetic  characters  to  repre- 
sent the  message  title  and  one  numeric  character  to  represent  the  action 
name.  The  TOS2  system  managers  have  replaced  their  DEVTOS-like  letter- 
letter-number  (LL#)  code  set  with  the  four-letter  (LLLL)  code  set  recom- 
mended by  the  TOS2  Format  Revision  Committee.  This  report  compares  the 
"learnability"  of  the  two  code  sets. 


METHOD 


Subjects 

The  test  subjects  were  20  enlisted  personnel  and  40  officers  from 
MASSTER  who  successfully  learned  one  of  four  message  reference  code 
sets. 3 a requirement  for  the  enlisted  men  was  that  they  have  an  Armed 
Forces  Qualification  Test  General  Technical  (GT)  score  of  90  or  higher; 
such  enlisted  men  presumably  would  be  more  representative  of  those  who 
might  function  as  TOS2  MIOD  (message  input-output  device)  operators  than 
would  a random  sample  of  Fort  Hood  enlisted  men.  A requirement  for  all 
subjects  was  that  they  must  not  have  had  prior  experience  in  using  the 
DEVTOS  message  reference  codes.  Officer  grades  ranged  from  01  to  05; 
enlisted  grades  ranged  from  E2  to  E7 . 

The  enlisted  personnel  were  randomly  assigned  to  learning  one  of 
the  two  G2  message  reference  code  sets.  Officers  were  randomly  assigned 
to  learning  one  of  these  two  G2  code  sets  or  to  one  of  the  two  G3  mes- 
sage reference  code  sets. 

Equipment  and  Workspace 

All  subjects  accomplished  their  learning  task,  one  at  a time,  at 
the  same  computer  terminal.  The  terminal  equipment  includes  a Hazeltine 
2000  cathode  ray  tube  (CRT)  and  a separate  keyboard  input  device.  A 
third  component,  a hard-copy  output  device,  was  not  used.  The  equipment 
was  situated  on  a table  in  a large,  well -lighted  room.  The  keyboard  was 
located  immediately  in  front  of  the  CRT,  and  either  item  could  be  moved 
to  suit  the  comfort  of  the  subject  seated  in  front  of  them. 


The  learning  performance  data  of  an  additional  six  officers  and  nine 
enlisted  personnel  was  incomplete  and,  hence,  not  usable.  In  nine  of 
these  cases  malfunctions  of  some  component  of  the  terminal/computer  dis- 
rupted and/or  terminated  operations.  In  three  cases  the  subject  did  not 
cooperate  properly,  and  in  another  three  cases  the  subject  was  called 
away  on  higher  priority  business. 
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The  computer  terminal  was  linked  to  an  IBM  370  computer  system  at 
Central  Texas  College,  Killeen,  Tex.  A computer  program  presented  the 
message  format  titles  and  appropriate  feedback  displays,  and  recorded/ 
compiled  data  regarding  response  times  and  errors.  Listings  of  these 
recorded  performance  measures  were  subsequently  used  in  performing  statis 
tical  analyses  of  the  performance  data. 


Procedure 


Instructions  to  Subjects.  Each  participant,  on  arrival,  was  greeted 
offered  coffee,  and  seated  at  a chair  facing  the  CRT.  The  experimenter 
or  an  assistant  sat  with  the  participant  to  demonstrate  keyboard  usage 
and  control  the  duration  of  exposure.  The  subject  of  tabular  informa- 
tion regarding  the  message  titles  was  displayed,  along  with  action  names 
and  their  associated  title  codes  and  action  codes,  which  jointly  com- 
prise the  message  reference  code.  For  uniformity  of  presentation,  in- 
structions were  read  to  each  subject.  Some  procedural  variation  was 
introduced  when  subjects  asked  questions  regarding  procedures  or  pur- 
poses; they  received  answers  that  others  did  not.  Performance  variation 
due  to  differences  among  the  presenters  of  instructions,  if  any  occurred, 
is  likewise  co-mingled  with  the  effects  of  the  main  independent  varia- 
bles: type  of  code,  G2  versus  G3  list,  and  rank  of  learner. 

The  instructions  and  the  four  sets  of  three  tables  describing  each 
of  the  four  code  sets  are  presented  in  Appendix  A. 

In  addition  to  describing  keyboard  usage,  CRT  display  content,  and 
timing  relationships,  the  instructions  indicated  that  subjects  were  to 
continue  at  the  task  until  they  made  a "perfect  pass,"  that  is,  the  sub- 
ject must  have  gone  through  the  entire  list  of  either  26  G2  message  for- 
mat titles  or  57  G3  titles  with  no  errors.  The  orderly  arrangement  of 
titles  during  the  first  pass  through  the  list  and  the  random  ordering 
of  the  items  on  subsequent  passes  were  described.  Each  subject  was  told 
the  purpose  of  the  experiment. 

As  the  subject  began  working,  the  experimenter  stayed  beside  him  to 
detect  any  misunderstanding  the  subject  might  have  concerning  how  to  re- 
spond. When  the  experimenter  judged  that  these  procedural  difficulties, 
if  any,  were  resolved,  he  moved  across  the  room  out  of  the  subject's 
vision. 

Display  Characteristics.  A given  message  title,  a hyphen,  and  the 
name  of  an  action  were  initially  displayed  in  capital  letters  on  the 
bottom  line  (of  15  lines  total)  of  the  CRT . This  stimulus  display  then 
appeared  rolled  upward  to  the  11th  line.  The  subject's  answer  (the 


Commercial  designations  are  given  only  in  the  interest  of  precision 
of  reporting.  Their  use  does  not  constitute  endorsement  by  the  U.S. 
Army  Research  Institute  or  the  U.S.  Army. 
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attempt  at  providing  the  message  reference  code)  appeared  on  the  12th 
line  as  the  subject  pressed  the  keys  to  enter  an  answer.  When  the  sub- 
ject pressed  the  "ESC"  key  to  transmit  a code-answer  to  the  computer, 
the  display  characters  shifted  up  two  lines,  and  a feedback  display  ap- 
peared on  the  empty  11th  line.  If  the  answer  (e.g.,  UL3)  was  correct, 
the  feedback  display  would  read  as  follows : 

UL3  IS  CORRECT 

If  UL3  was  wrong  and  the  subject  should  have  answered  UL2,  the  feedback 
display  would  read  thus : 


•ERROR* 

UL2  IS  CORRECT 

The  ‘ERROR*  appeared  first  on  line  11;  all  displayed  characters  were 
shifted  up  a line;  then  UL2  IS  CORRECT  was  displayed  on  the  now-vacant 
11th  line. 

This  technique  of  shifting  or  rolling  displayed  lines  upward  was 
used  so  that  the  subject  could  anticipate  coming  events.  Inserted  five 
lines  before  each  title  display  was  this  instruction: 

ENTER  ANSWER,  THEN  PRESS  "ESC"  TO  CONTINUE 

The  separations  among  displayed  lines  were  arranged  so  that  when  a new 
title  rolled  up  to  line  11,  the  feedback  display  for  the  previous  title 
disappeared  at  the  top  of  the  CRT.  The  following  two  display  lines  ap- 
peared at  the  end  of  a pass  through  the  list  of  titles  wherein  one  or 
more  incorrect  answers  had  been  given: 

YOU  HAVE  COMPLETED  PASS 
YOU  ARE  REQUIRED  TO  CONTINUE 

When  the  subject  met  the  criterion  of  learning  by  making  a perfect  pass, 
these  two  display  lines  appeared: 

YOU  HAVE  COMPLETED  PASS 
YOU  HAVE  FINISHED 


Experimental  Variables 


Independent  Variables . Three  independent  variables  were  used  in 
this  experiment;  the  one  called  "code  type"  or  "type  of  code"  is  of  pri- 
mary interest.  One  of  the  two  types  of  codes  used  consists  of  two  al- 
phabetic characters  followed  by  a digit,  i.e.,  the  old  DEVTOS  LL#  code. 
The  two  letters  represent  the  title  of  a message  category,  and  the  digit 
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represents  the  type  of  action  the  user  desires  to  take  with  respect  to 
that  category.  Note,  however,  that  in  this  experiment,  message  formats 
were  not  displayed — only  the  names/titles  of  the  message  categories  and 
associated  permissible  actions. 

The  second  type  of  code  used  is  a four-letter  code.  The  first  three 
letters  represent  the  title  of  a message  category,  and  the  fourth  letter 
represents  the  type  of  action  desired. 

The  "rules"  or  schemes  by  which  the  two  code  sets  are  generated  are 
different.  It  is  judged  that  the  rules  regarding  the  new  four-letter 
code  are  simpler  to  state  (see  Tables  1 and  2) , partly  because  there  are 
fewer  exceptions.  An  individual  learning  the  new  LLLL  codes  is  typically 
forming  acronyms  by  using  the  first  letter  of  each  word.  Most  of  the 
difficulty  in  learning  the  LLLL  codes  should  arise  when  the  message  cate- 
gory title  contains  other  than  three  words.  In  the  old  LL#  code,  the 
first  letter  sorts  between  friendly,  enemy,  and  "common"  message  cate- 
gories, with  an  exception  included  ("F"  for  aircraft-related  message 
categories) . The  second  letter  is  arbitrarily  assigned  to  the  messages 
within  each  of  the  friendly  or  enemy  categories,  but  it  is  always  an  "A" 
for  the  common  messages.  For  the  action  code,  the  numbers  1 through  6 
are  used  for  the  six  permitted  actions,  but  exceptions  arise  with  the 
common  messages  where  0,  8,  and  9 replace  the  1,  3,  and  4 to  enable  the 
computer  program  to  discriminate  among  actions.  The  preceding  descrip- 
tion of  the  rules  by  which  the  two  code  sets  are  generated  is  incomplete, 
but  it  helps  to  show  why  the  four-letter  code  was  recommended  for  incor- 
poration in  the  TO S2  system  and  why  it  was  predicted  that  learning  the 
four-letter  code  would  be  easier  than  learning  the  LL#  code. 

A second  independent  variable  sorts  learners  into  two  categories: 
officers  and  enlisted  men.  Both  may  be  employed  in  roles  wherein,  among 
component  tasks,  they  must  use  the  message  reference  code. 

Subjects  learned  either  the  codes  for  the  G2  message,  plus  action 
terms,  or  the  codes  for  the  G3  terms;  thus  G2  versus  G3  becomes  the 
third  independent  variable  whose  effects  on  performance  can  be  assessed. 
This  assessment  can  be  made  only  for  the  officers,  however,  as  the  avail- 
able number  of  enlisted  men  did  not  allow  assigning  them  to  the  G3  code 
learning  task. 

Dependent  Variables . The  percentage  of  message  reference  codes 
input  with  one  or  more  characters  in  error  (or  not  supplied)  is  judged 
to  be  the  most  pertinent  measure  of  learning  performance  (dependent  var- 
iable) in  this  experiment.  Another  dependent  variable  is  the  number  of 
passes  through  the  code  list  that  each  learner  made  in  reaching  the 
success  criterion  of  a perfect  (error-free)  pass.  This  measure  counts 
a pass  with  one  error  the  same  as  a pass  with  many  errors  and  therefore 
is  a less  sensitive  measure  of  performance  than  the  error  rate  measure 
mentioned  above;  however,  the  two  measures  should  be  positively 
correlated . 
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Table 


Message  Titles  with  Old  (LL#i 


G2  message  titles 


Enemy  Unit  Status 

Enemy  Situation  Data 

(Enemy)  Intelligence  Summary 

(Enemy)  Intelligence  Working  File 

Enemy  Situation  Data  Base  Index 

Relay3 

Named  Area  of  Interest3 

a 

Standing  Request  for  Information  File 


G3  message  titles 


Task  Organization 
Task  Force 

Tactical  Dispositions 

Unit  Disposition,  One  Coordinate 

Unit  Disposition,  One  Unit 

Unit  Disposition,  General 

Air  Control  Measure 

Operations  Journal 

Situation  Report 

Pending  Change 

Spot  Report 

Aircraft  Available 

— 

a 

These  messages  are  also  used  for  G3 
G3  message  titles  as  well. 

r 

I 


Table  2 


Action  Names  with  Old  and  New  Codes 


Action 

code 

Action  name 

01da 

New 

Add 

1 

A 

Change 

2 

C 

Delete 

3 

D 

Query 

4 

Q 

Special  processing 

5 

p 

Establish  standing  request  for  information 

6 

S 

a 

Plus  these  special  cases: 


1.  Use  an  action  code  of  0 to  add  a Relay  message. 

2.  use  an  action  code  of  8 to  delete  an  "SRI  File"  message. 

3.  Use  an  action  code  of  9 to  query  the  "SRI  File." 


Instructions  to  the  subjects  did  not  suggest  that  they  emphasize 
speed  of  response.  Explicit  emphasis  such  as  "Be  accurate,  not  fast" 
was  not  given.  However,  the  weight  of  the  instructions  is  thought  to 
favor  accuracy  and  to  ignore  speed.  At  any  rate,  the  dependent  variable 
of  time  to  input  a message  reference  code  was  computed  for  each  learner, 
averaged  over  the  26  or  57  code  inputs  for  each  pass , and  over  all  passes . 

Supplementary  analyses  of  error  rates  were  made  for  the  action  code 
alone  (last  character) , for  the  message  title  code  (first  two  or  first 
three  characters) , and  for  each  character  position.  This  was  done  to 
determine  if  the  arbitrariness  in  the  second  letter’s  assignment  and 
the  exceptions  in  the  action  code  (previously  pointed  out  regarding  the 
LL#  code)  inflated  the  error  rates  in  comparison  to  corresponding  error 
rate  scores  for  the  four-letter  code. 


RESULTS  AND  DISCUSSION 


Effects  of  Type  of  Code 

LLLL  codes  were  clearly  superior  to  LL#  codes  in  that  subjects 
learned  the  LLLL  codes  in  fewer  passes  and  with  a lower  rate  of  errors. 
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Number  of  Passes  to  Criterion.  Subjects  learning  the  LLLL  code 
achieved  the  goal  of  successful  code  learning  in  fewer  passes,  on  the 
average,  than  did  subjects  learning  the  LL#  code  (see  Table  3) . This 
result  was  obtained  for  enlisted  personnel  (5.2  versus  9.2  passes)  and 
for  officers  (3.8  versus  6.8  passes)  working  on  the  G2  message  reference 
codes  and  again  for  the  officers  (2.9  versus  4.7  passes)  working  on  the 
G3  message  reference  codes.  Among  these  three  comparisons  of  LLLL  ver- 
sus LL#,  there  is  a strong  consistency  of  relationship;  i.e.,  the  mean 
number  of  passes  needed  to  learn  the  LLLL  code  is,  respectively,  57%, 

56%,  and  62%  of  the  number  of  passes  needed  to  learn  the  LL#  code. 

Number  of  passes  is,  of  course,  higher  correlated  with  total  time  to 
learn. 

Statistical  analysis  of  these  results  indicates  that  the  differences 
favoring  learning  of  the  four-letter  code  are  statistically  significant 
for  each  of  the  three  comparisons  (p  < .01;  p < .01;  p < .05  respectively) 
noted  above.  Analysis  of  variance  tables  relating  to  these  and  other 
dependent  variables  are  presented  in  Appendix  B. 

Percentage  of  Errors.  The  mean  percentage  of  errors  (responses 
containing  at  least  one  error)  made  by  each  of  the  six  groups  of  sub- 
jects is  shown  in  Table  3 and  Figure  1.  Those  learning  the  four-letter 
message  reference  codes  had  smaller  percentages  of  errors  than  did  those 
learning  the  LL#  codes.  As  with  number  of  passes,  this  result  was  ob- 
tained for  the  two  groups  of  enlisted  personnel  (13.2%  versus  29.1%)  and 
the  two  groups  of  officers  (10.6%  versus  21.4%)  working  on  the  G2  message 
reference  codes,  and  also  for  the  two  groups  of  officers  (6.5%  versus 
15.1%)  who  learned  the  G3  message  reference  codes.  All  three  differences 
are  statistically  significant  (p  < .01  in  each  case) . The  error  rates 
associated  with  the  LLLL  codes  are  45%,  50%,  and  43%  of  the  size  of  the 
error  rates  associated  with  the  learning  of  the  LL#  codes  for  the  three 
comparisons,  respectively.  The  consistency  of  these  error  rate  relation- 
ships (45%  and  50%)  for  the  enlisted  men  and  officers  working  with  the 
G2  message  titles  is  reflected  in  the  analysis  finding  of  a nonsignifi- 
cant and  very  small  interaction  of  learner's  rank  with  type  of  code. 

Time  To  Input  a Message  Reference  Code.  Results  for  this  dependent 
variable  are  shown  in  Table  3.  The  mean  time  to  input  a message  refer- 
ence code  (averaged  across  inputs  for  a given  learner,  and  then  averaged 
across  learners  within  a group)  was  not  significantly  nor  practically 
influenced  by  code  type:  i.e.,  LLLL  code  versus  LL#  code.  This  result 
appears  to  be  consistent  with  the  lack  of  emphasis  on  inputting  code  re- 
sponses quickly.  Recall  also  that  subjects  had  20  seconds  to  input  each 
code  response;  this  was  13  to  14  seconds  more  than  the  average  time 
taken.  No  one  complained  that  this  length  of  time  was  a source  of  pres- 
sure, irritation,  or  other  concern;  neither  did  the  monitors  judge  it  to 
be  such. 


Table  3 


Error 

Rate, 

Passes , 

and  Code 

Input  Time 

Rank 

List 

No.  of 
items 

Code 

type 

No.  of 
subjects 

Mean  % 

errors 

Mean  no . of 
passes 

Mean  input 
time  (sec.) 

EM 

G2 

26 

LL# 

10 

29.1 

9.2 

6.50 

EM 

G2 

26 

LLLL 

13 

13.2 

5.2 

6.44 

OFF 

G2 

26 

LL# 

10 

21.4 

6.8 

6.28 

OFF 

G2 

26 

LLLL 

10 

10.5 

3.8 

6.53 

OFF 

G3 

57 

LL# 

9 

15.1 

4.7 

5.64 

OFF 

G3 

57 

LLLL 

8 

6.5 

2.9 

5.54 

Note . 

Errors 

refers 

to  codes  input 

with  at 

least  one  character  incor- 

rectly  supplied  or  not  supplied. 


Officer-Enlisted  Personnel  Differences 


The  median  GT  score  for  the  enlisted  personnel  participating  in 
this  experiment  was  124.  Despite  their  high  GT  score,  the  enlisted 
personnel  took  more  passes  in  learning  the  LL#  codes  (9.2  versus  6.8 
passes)  and  in  learning  the  LLLL  codes  (5.2  versus  3.8  passes)  than 
did  the  officers.  Analysis  of  variance  of  the  2x2  factorial  design 
(2  ranks  x 2 code  types  on  G2  messages)  indicated  that  the  overall  dif- 
ference between  the  two  groups'  performances  was  statistically  signifi- 
cant ( .01  < p < .05) . 

In  terms  of  mean  percentage  of  errors  made  while  learning  the  G2 
message  reference  codes,  EM  had  a higher  error  rate  in  learning  the 
LL#  codes  (29.1%  versus  21.4%)  and  in  learning  the  LLLL  codes  (j.3.2% 
versus  10.6%)  than  did  the  officers.  The  EM-officer  overall  error  rate 
comparison  (21.14%  versus  16.0%)  was  associated  with  a probability  of 
.053,  which  just  barely  fails  to  meet  the  null  hypothesis  rejection 
level  of  .05. 

Given  either  a larger  sample  size  and/or  a lower  distribution  of 
EM's  GT  scores,  these  error  rate  differences  would  probably  have  been 
significant  at  the  .05  level.  Also,  the  gap  between  officer  and  EM 
error  rates  is  greater  for  the  more  difficult  LL#  codes  than  for  the 
LLLL  codes.  If  LLLL  message  reference  codes  do  close  the  gap  between 
officer  and  EM  performance,  this  might  be  exploited  beneficially  in 
future  TOS  operations. 
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Mean  °7°  Errors 


/ 


Observed  differences  between  officers  and  enlisted  personnel  for 
mean  time  to  input  a message  reference  code  averaged  only  .1  and  .2 
seconds  for  the  two  comparisons.  These  differences  were  far  from  being 
statistically  or  practically  significant. 


G2  Versus  G3  Differences 


Subjects  learned  message  reference  codes  related  to  G3  operations 
in  fewer  passes  than  they  learned  the  G2  codes.  For  the  LL#  type  of 
code,  the  comparison  is  4.7  passes  versus  6.8  passes.  For  the  LLLL 
codes,  those  applying  to  G3  were  learned  in  2.9  passes,  on  the  average, 
versus  3.8  passes  for  the  G2  list. 

Error  rate  data  also  favored  the  learning  of  codes  related  to  G3 
operations  over  codes  related  to  G2  operations.  The  overall  comparison 
is  10.8%  errors  on  the  G3  lists  versus  16.0%  errors  on  the  G2  lists. 

This  difference  is  statistically  significant  (p  < .01) . For  the  LL# 
codes,  officers  working  on  the  G3  list  made  15.1%  errors  compared  to 
21.4%  errors  for  the  officers  working  on  the  G2  list.  For  the  LLLL 
codes,  there  were  10.8%  G3  errors  versus  16.0%  G2  errors. 

Overall,  the  mean  time  to  input  a code  while  learning  was  0.87 
seconds  faster  for  the  G3  list  of  codes  (5.60  seconds)  than  for  the  G2 
list  (6.47  seconds).  The  probability  of  obtaining  a time  difference 
this  large  or  larger  by  chance,  under  the  hypothesis  of  no  difference, 
falls  short  (p  = .08)  of  the  rejection  level  of  .05.  If  reliable,  the 
smaller  mean  time  to  input  a G3  message  reference  code  may  be  attributed 
to  the  fact  that  the  G3  list  has  a larger  number  and  proportion  of  mes- 
sage titles  that  could  be  responded  to  correctly;  hence,  the  overall 
mean  time  to  input  a message  reference  code  is  shorter. 


Errors  by  Character  Position 

To  gain  a better  understanding  of  the  sources  of  error  and  diffi- 
culty in  learning  the  LL#  and  LLLL  codes,  an  analysis  of  errors  by  char- 
acter position  was  made  for  each  officer  within  each  of  the  four  combina- 
tions of  list  type  (G2  and  G3)  and  code  type.  The  results  of  this 
analysis  are  presented  in  Appendix  C.  (The  actual  individual  performance 
data  are  shown  in  Appendix  D.) 


Alternative  Message  Reference  Code  Schemes 

As  noted,  subjects  learned  the  LLLL  code  with  about  half  the  error 
percentage  associated  with  learning  the  LL#  code.  However,  the  analysis 
of  errors  by  character  position  (Appendix  C)  revealed  that  seven  of  the 
new  code's  characters  were  associated  with  excessively  high  error  rates 
in  the  learning  situation  used  in  the  experiment.  Thus,  it  seems 
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possible  to  further  reduce  error  rates  in  learning  and  using  message 
reference  codes.  Four  alternative  approaches  to  achieving  this  addi- 
tional error  reduction  are  described  and  discussed  in  the  paragraphs 
that  follow. 

Approach  A:  Partial  Revision.  If  only  those  message  titles,  ac- 
tion names,  and  code  characters  associated  with  excessively  high  error 
rates  (10%  or  more)  were  revised,  a substantial  further  reduction  in 
error  rates  in  learning  the  message  reference  codes  could  be  expected. 
Table  4 shows  the  message  reference  titles  (split  into  message  titles 
and  action  names)  and  code  characters  for  both  G2  and  G3  operations. 

An  asterisk  indicates  that  a revision  has  been  made.  Note  that  the 
message  titles  do  not  all  contain  exactly  three  words.  Consequently 
the  three-character  code  associated  with  the  message  titles  cannot  be 
completely  acronymic.  This  means  that  the  procedure  by  which  the  code 
has  been  generated  cannot  be  explained  to  a potential  learner  with  a 
fairly  simple,  easy-to-remember  statement.  However,  approaches  B,  C, 
and  D all  have  the  advantage  of  being  easy  to  explain  to  a learner. 

Approach  B:  Three-Word  Message  Titles,  Three-Letter  Acronyms. 

In  this  approach,  all  message  titles  are  forced  to  have  exactly  three 
words  so  that  the  first  letter  of  each  word  (in  the  message  title)  can 
be  used  to  form  an  acronym.  This,  in  turn,  permits  using  a sentence 
such  as  the  following  to  describe  to  a learner  how  to  determine  the 
code  for  any  message  title  plus  action  name  combination:  "The  message 
reference  code  consists  of  the  first  letter  of  each  word  in  the  message 
title  followed  by  the  first  letter  of  the  first  word  in  the  action  name." 
Table  5 displays  the  message  reference  titles  and  codes  in  the  same 
format  as  Table  4.  The  use  of  this  approach  raises  this  question — 

Does  the  availability  of  the  exact  three-letter  acronym  "rule"  compensate 
for  the  many  changes  in  the  message  titles  that  are  involved? 

Approach  C:  Variable-Length  Acronymic  Code.  This  approach  re- 
quires presumably  minor  computer  program  revision  so  that  the  action 
officer  may  input  message  reference  codes  of  2,  3,  or  4 characters  ac- 
cording to  the  presence  in  the  message  titles  of  1,  2,  or  3 words. 

(The  action  code  adds  the  additional  character,  of  course.)  The  explana- 
tion given  a learner  of  this  message  reference  code  could  be  as  follows : 
"The  code  consists  of  the  first  letter  of  each  word  in  the  message  title, 
then  the  first  letter  of  the  first  word  of  the  action  name." 

Implementation  of  this  code  requires  message  titles  to  consist  of 
three  words  or  less  with  no  ambiguities;  e.g.,  two  titles  generating 
the  same  code.  Table  6 displays  the  message  titles,  action  names,  and 
code  letters  involved  in  this  variable-length,  acronymic  code. 

This  approach  involves  changes  to  8 message  titles,  compared  to 
16  title  changes  for  approach  B. 
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Table  4 


A Partial  Revision  of  the  LLLL  Code  and  Titles 


Message  title 

Code  letters 

Task  Organization 

UTO 

Task  Force 

UTF 

Tactical  Dispositions 

UTD 

Unit  Disposition,  One  Coordinate 

UDC 

Unit  Disposition,  One  Unit 

UDU 

Unit  Disposition,  General 

UDG 

Air  Control  Measure 

ACM 

Operations  Journal 

UOJ 

SITREP* 

SIT* 

Pending  Change 

UP  C 

Spot  Report 

USR 

Aircraft  Available 

UAA* 

Common  Relay  Message* 

CRM* 

Named  Area  of  Interest 

NAI 

Standing  Request  for  Information* 

SRI 

Enemy  Unit  Status 

EUS 

Enemy  Situation  Data 

ESD 

(Enemy)  Intelligence  Summary 

EIS  * 

Intelligence  Work  File* 

IWF* 

Situation  Data  Index* 

SDI* 

Action  name 

Code  letter 

Add 

A 

Change 

C 

Delete 

D 

Query 

Q 

Special  Processing 

s* 

Establish  Standing  Request  for  Information 

E* 

•Changed  from  current  version. 
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Table  5 


Three-Word  Message  Titles  with  Three-Letter 
Acronymic  Codes 


Message  title 

Code  letters 

Unit  Task  Organization* 

UTO 

Unit  Task  Force* 

UTF 

Unit  Tactical  Dispositions* 

UTD 

Unit  Disposition,  Coordinate* 

UDC 

Unit  Disposition,  Unit* 

UDU 

Unit  Disposition,  General 

UDG 

Air  Control  Measure 

ACM 

Unit  Operations  Journal* 

UOJ 

Tactical  Situation  Report* 

TSR* 

Unit  Pending  Change* 

UPC 

Unit  Spot  Report* 

USR* 

Friendly  Aircraft  Available* 

FAA* 

Common  Relay  Message* 

CRM* 

Named  Interest  Area* 

NIA* 

Standing  Information  Request* 

SIR* 

Enemy  Unit  Status 

EUS 

Enemy  Situation  Data 

ESD 

Enemy  Intelligence  Summary* 

EIS* 

Intelligence  Work  File* 

IWF* 

Situation  Data  Index* 

SDI* 

Action  name 

Code  letter 

Add 

A 

Change 

C 

Delete 

D 

Query 

Q 

Special  Processing 

S* 

Establish  Standing  Information  Request* 

E* 

♦Changed  from  current  version. 
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Table  6 


Message  Titles,  Action  Names,  and  Code  Letters 
Required  by  a Variable  Length  Acronymio  Code 


Message  title 

Code  letter (s) 

Task  Organization 

TO* 

Task  Force 

TF* 

Tactical  Dispositions 

TD* 

Unit  Disposition,  Coordinate* 

UDC 

Unit  Disposition,  Unit* 

UDU 

Unit  Disposition,  General 

UDG 

Air  Control  Measure 

ACM 

Operations  Journal 

OJ* 

STTREP* 

S* 

Fending  Change 

PC* 

Spot  Report 

SR* 

Aircraft  Available 

AA* 

Relay 

R* 

Named  Interest  Area* 

NIA* 

Standing  Information  Request* 

SIR* 

Enemy  Unit  Status 

EUS 

Enemy  Situation  Data 

ESD 

Intelligence  Summary* 

IS* 

or  INTSUM* 

I* 

Intelligence  Work  File* 

IWF* 

Situation  Data  Index* 

SDI  * 

Action  name 

Code  letter 

Add 

A 

Change 

C 

Delete 

D 

Query 

Q 

Special  Processing 

S* 

Establish  Standing  Information  Request* 

E* 

♦Changed  from  current  version. 
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Approach  D:  A Three-Letter  Code  Based  on  First  and  Last  Words  of 
Message  Title.  Approach  D could  be  explained  with  a statement  such  as 
"The  code  consists  of  the  first  two  letters  of  the  first  word  of  the 
message  title,  plus  the  first  letter  of  the  last  word,  followed  by  the 
first  letter  of  the  first  word  of  the  action  name."  This  approach  does 
not  employ  an  exact  acronym,  although  the  code  letters  form  a partial 
or  incomplete  acronym.  (Webster's  Seventh  New  Collegiate  Dictionary 
defines  acronym  as  "a  word  (as  radar,  snafu)  formed  from  the  initial 
letter  or  letters  of  each  of  the  successive  parts  or  major  parts  of  a 
compound  term.")  This  approach  requires  the  change  of  only  one  message 
title.  Table  7 displays  the  message  titles,  action  names,  and  code 
letters  involved  in  this  three-letter,  incomplete  acronymic  code. 

Which  Approach  Is  Best?  It  is  expected  that  all  four  approaches 
to  revising  or  designing  a message  reference  code  would  result  in  fur- 
ther reduction  of  error  rates;  however,  approaches  B,  C,  and  D are 
believed  to  be  capable  of  bringing  about  more  error  reduction  than  the 
partial  revision  described  in  approach  A.  A further  experiment  is 
needed  to  determine  which  of  the  three  approaches  is  best  at  reducing 
errors;  such  an  experiment  is  being  prepared. 

CONCLUSIONS  AND  RECOMMENDATIONS 

It  may  be  reliably  concluded  that  the  four-letter  message  refer- 
ence code  set  recommended  for  use  with  the  TOS^  can  be  learned  more 
quickly  and  with  less  error  than  the  alternative  two-letter,  one-number 
code  set  that  had  been  used  by  DEVTOS  personnel . 

It  is  recommended  that  consideration  be  given  to  revising  the  cur- 
rent four-letter  code  version  in  accordance  with  Table  4 or  changing  it 
to  one  of  the  three  versions  shown  in  Tables  5,  6,  and  7.  Adoption  of 
any  of  these  four  versions  should  further  reduce  the  number  of  errors 
m<jde  by  code  learners. 


Table  7 


Message  Titles,  Action  Names,  and  Code  Letters 
Required  by  a Three-Letter  Partially  Acronymic  Code 


Message  title 

Code  letters 

Task  Organization 

TAO* 

Task  Force 

TAF* 

Tactical  Dispositions 

TAD* 

Unit  Disposition,  One  Coordinate 

UNC 

Unit  Disposition,  One  Unit 

UNU 

Unit  Disposition,  General 

UNG 

Air  Control  Measure 

AIM* 

Operations  Journal 

OPJ* 

Situation  Report 

SIR* 

Pending  Change 

PEC* 

Spot  Report 

SPR* 

Aircraft  Available 

AIA* 

Relay  Message* 

REM* 

Named  Area  of  Interest 

NAI 

Standing  Request  for  Information  File 

STF* 

Enemy  Unit  Status 

ENS* 

Enemy  Situation  Data 

END* 

Intelligence  Summary 

INS* 

Intelligence  Work  File 

INF* 

Enemy  Situation  Data  Base  Index 

ENI* 

Action  name 

Code  letter 

Add 

A 

Change 

C 

Delete 

D 

Query 

Q 

Special  Processing 

S* 

Establish  Standing  Request  for  Information 

E* 

‘Changed  from  current  version. 


APPENDIX  A 

INFORMATION  AND  INSTRUCTIONS  FOR  SUBJECTS 

Information  and  Instructions  Read  to  Subjects 

MASSTER,  the  Computer  Systems  Command,  the  Army  Tactical  Data  Systems 
Project,  and  the  Army  Research  Institute  have  been  experimentally  using 
computer  systems  to  assist  in  message  processing.  For  this  test  we  have 
taken  one  of  the  G2  or  G3  officer's  sub-tasks,  and  will  have  you  learn 
it. 

The  G3  (or  G2)  must  use  a message  reference  code  to  tell  the 
computer  which  of  6 actions  he  wants  to  take  on  any  of  the  15  (or  8) 
categories  of  messages  that  he  handles.  (Show  Tables  1 and  2) 

In  table  a you'll  see  the  names  of  message  categories  and  the  code 
that  stand  for  each. 

In  table  b you’ll  see  the  names  of  the  six  actions  that  can  be  taken, 
along  with  the  single  character  action  code. 

The  message  reference  code  which  you  will  be  learning  during  this 
experiment  is  formed  by  adding  these  two  codes,  as  shown  in  table  c. 

Would  you  please  read  aloud  the  names  and  codes  from  table  a and  b 
so  that  I'll  know  you  have  paid  some  attention  to  each  one. 

(After  he's  read  aloud,  take  away  the  tables.) 

After  we  get  the  program  going,  you  will  see  a message  and  action 
name  appear  here  at  the  bottom  of  the  CRT.  It  will  roll  up  (move  your 
finger  up  4 times)  one,  two,  three  four  lines  and  stop.  After  it 
stops  rolling,  you  will  have  20  seconds  to  type  in  the  message  reference 
code.  What  you  type  will  appear  on  the  CRT.  After  you've  typed  in  what 


you  think  the  code  is,  then  hit  this  "ESC"  button  to  tell  the  program  that 
you  are  finished  making  your  response. 

The  program  will  then  give  you  feedback.  If  your  answer  is  correct, 
it  will  repeat  your  answer  and  say  it  is  correct.  If  your  answer  is 
wrong,  the  v/ord  "ERROR"  will  be  displayed  and  then  the  correct  answer 
will  be  displayed.  This  feedback  will  last  for  5 seconds  and  then  the 
program  will  go  on  to  the  next  item. 

If  you  see  that  you've  typed  in  something  wrong  and  haven't  yet  hit 
the  "ESC"  button,  then  here's  how  to  correct  it.  You  must  replace  the 
wrong  character  and  all  characters  to  the  rignt.  Tell  the  program  how 
many  characters  you're  going  to  replace  by  pressing  the  shift  key  and 
hitting  this  arrow  (upper  case  0).  Then  type  in  the  correction  and 
send  it. 

There  are  26  (57)  titles  in  the  list  you  will  be  learning.  On  the 
first  pass  through  the  list  the  titles  will  be  displayed  in  this  order 
(point  at  table  c).  On  the  second  and  later  passes  the  titles  will  appear 
in  a random  order.  You  will  be  finished  when  you  make  a perfect  pass; 
i.e.,  go  through  the  whole  list  without  making  an  error. 

The  purpose  of  this  experiment  is  to  get  performance  data  on  how  easy 
or  difficult  various  message  reference  codes  are  to  learn.  When  the 
experiment  is  over,  we'll  inform  the  ARTADS  people  of  our  findings  and 
they  will  make  use  of  them  in  a new  computer  system  that  will  be 
coming  here  in  about  a year. 

Tables  Shown  to  Subjects 

On  the  following  pages  are  the  tables  that  were  shown  to  subjects 
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as  part  of  their  information  and  instructions.  A given  subject  saw  only 
three  tables  a,  b,  and  c.  Which  set  of  three  he  was  shown  - set  1,  2, 

3 or  4 - depended  on  which  code  (LL#  or  LLLL)  and  which  area  of  operations 
(G2  or  G3)  he  was  assigned  to  learn. 


Table  A-la 


Message  Titles  And  Their  Abbreviations,  For  G2  Functions 


No. 

Message  Title 

Message  Title 
Abbreviations 

T 

Enemy  Unit  Status 

TUS 

2 

Enemy  Situation  Data 

ESD 

3 

(Enemy)  Intelligence  Summary 

EIN 

4 

(Enemy)  Intelligence  Workinq  File 

EWF 

5 

Enemy  Situation  Data  Base  Index 

EDX 

6* 

Relay 

REL 

7* 

Named  Area  of  Interest 

NAI 

8* 

Standing  Reouest  for  Information  File 

SPI 

*These 

types  of  messages  are  also  used  for  G-3 

functions. 

Table  A-lb 

Types  of  Actions  and  the  Letter  Code  for  Fach 


Action  Desired  Regarding  Message Action  Code 

ABE  (to  data  base)  " — 

CHANGE  (the  data  base)  C 

DELETE  (from  data  base)  D 

QUERY  (the  data  base)  Q 

SPECIAL  PROCESSING  (of  some  message  tvpes)  P 

ESTABLISH  STANDING  REQUEST  FOR  INFORMATION  S 
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Table, A-lc 


Message  Code  List  To  Be  Learned  In  Response  To  Message  Title 
And  Type  Of  Action  In  The  Area  Of  u2  Functions 


No. 

Messaqe  Title  And  Action 

Messaqe 

1 

Enemy  Unit  Status-ADD 

2 

Enemy  Unit  Status-DELETE 

EUSO 

3 

Enemy  Unit  Status-QUERY 

EIJSO 

4 

Enemy  Situation  Data-ADD 

ESDA 

5 

Enemy  Situation  Data-CHANGE 

ESDC 

6 

Enemy  Situation  Data-DELETE 

ESDD 

7 

Enemy  Situation  Data-OUERY 

ESDO 

8 

Enemy  Situation  Data-ESTABLISW  SRI 

ESDS 

9 

(Enemy)  Intelligence  Summary-APD 

EINA 

10 

(Enemy)  Intelligence  Summary-CHAf.'GE 

EINC 

11 

(Enemy)  Intelligence  Sunmary-OELETE 

El  ND 

12 

(Enemy)  Intelligence  Summarv-SPECIAL  PROCESSING 

EINP 

13 

(Enemy)  Intelligence  Work  File-ADD 

EWFA 

14 

(Enemy)  Intelligence  Work  File-CFANGE 

EWFC 

15 

(Enemy)  Intelligence  Work  File-CELETE 

EWFD 

16 

(Enemy)  Intelligence  Work  File-QUERY 

EWFO 

17 

Enemy  Situation  Data  Base  Index-ADD 

EDXA 

18 

Enemy  Situation  Oata  Base  Index-CHANGE 

EOXC 

19 

Enemy  Situation  Data  Base  Index-OUERY 

EDXO 

20 

Relay-ADD 

RFLA 

21 

Named  Area  of  Interest-ADD 

NAIA 

22 

Named  Area  of  Interest-CIIANGE 

NAIC 

23 

Named  Area  of  Interest-DELETE 

NAIO 

24 

Named  Area  of  Interest-OUERY 

NAIO 

25 

Standing  Request  For  Information  File-DELETE 

SRIO 

26 

Standing  Request  For  Information  File-QUERY 

SRIO 

Code 
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Table  A-2a 


Message  Titles  And  Their  Abbreviations,  For  G2  Functions 


No. 

Messaqe  Title 

Messaqe  Title 
Abbreviation 

1 

Enemy  Unit  Status 

EA 

2 

Enemy  Situation  Data 

EC 

3 

(Enemy)  Intelligence  Summary 

ED 

4 

(Enemy)  Intelligence  Uorkino  File 

EE 

5 

Enemy  Situation  Oata  Base  Index 

EG 

6* 

Relay 

AA 

7* 

Named  Area  of  Interest 

AA 

8* 

Standing  Request  For  Information  File 

AA 

*Yhese  types  of  messages  are  also  used  for  G3  functions. 


Table  A-2b 


Types  Of  Actions  And  The  Numerical  Code  For  Each 


Action  Desired  Regarding  Message Action  Code 

AbO  (to  data  base)  1 
CHANGE  (the  data  base)  2 
DELETE  (from  data  base)  3 
QUERY  (the  data  base)  4 
SPECIAL  PROCESSING  (of  seme  message  tvpes)  5 
ESTABLISH  STANDING  REOl'EST  FOR  INFORMATION  6 


Special  Cases:  (T)  Only  an  Action  Code  of  0 may  be  used  with  the  relav 

message. 

(2)  Use  an  Action  Code  of  8 to  delete  an  SRI  File  message. 

(3)  Use  an  Action  Code  of  9 to  query  the  SRI  File. 


Table  A-2c 

Message  Code  List  To  Be  Learned  In  Response  To  Message  Title 
And  Type  Of  Action  In  The  Area  Of  02  Functions 


Message  Title  And  Action 

Enemy  Unit  Status-ADb 
Enemy  Unit  Status-DELETE 
Enemy  Unit  Status-QUERY 

Enemy  Situation  Data-ADD 
Enemy  Situation  Data-CHANGE 
Enemy  Situation  Data-OCLETE 
Enemy  Situation  Data-OUERY 
Enemy  Situation  Data-ESTAGLISH  SRI 

(Enemy)  Intelligence  Surma ry- ADD 

(Enemy)  Intelligence  Sumrnarv-CHANGE 

(Enemy)  Intelligence  Summary-DELETE 

(Enemy)  Intelligence  Summary-SPECIAL  PROCESSING 

(Enemy)  Intelligence  Work  File-ADD 
(Enemy)  Intelligence  Work  File-CPANGF 
(Enemy)  Intelligence  Work  File-DELETF 
(Enemy)  Intelligence  Work  File-Ql'ERv 

Enemy  Situation  Data  Ease  Index-APD 
Enemy  Situation  Data  Base  Index-CHAMG.E 
Enemy  Situation  Data  Case  Index-OCCRY 

Relay-ADD 

Named  Area  of  Interest-ADD 
Named  Area  of  Interest-CWANGE 
Named  Area  of  Interest-DELETE 
Named  Area  of  Interest-QUERY 

Standing  Request  For  Information  File-DELETE 
Standing  Request  For  Information  File-OUERY 


Messaqe  Code 

TF1 

EA  3 
EA  4 


Table  A- 3a 


Message  Titles  And  Their  Abbreviations,  For  G3  Functions 

Message  Title 


No. 

Messaqe  Title 

Abbreviation 

1 

Task  Organization 

UTO 

2 

Task  Force 

UTF 

3 

Tactical  Dispositions 

UTD 

4 

Unit  Disposition,  One  Coordinate 

UDC 

5 

Unit  Disposition,  One  Unit 

UDU 

6 

Unit  Disposition,  General 

UDC, 

7 

Air  Control  Measure 

ACM 

8 

Operations  Journal 

UOd 

9 

Situation  Report 

US  I 

10 

Pending  Change 

UPC 

11 

Spot  Report 

USR 

12 

Aircraft  Available 

AAV 

13 

Relay 

REL 

14 

Named  Area  of  Interest 

NAI 

15 

Standing  Request  for  Information 

SRI 

Table  A-3b 

Types  of  Action  and  the  Letter  Code 

Action  Desired  P.eoardina  Message 

for  Each 

Action  Code 

ADD  (to  data  base) 

~T~ 

CHANGE  (the  data  base) 

C 

DELETE  (from  data  base) 

D 

QUERY  (the  data  base) 

Q 

SPECIAL  PROCESSING  (of  some  message  tvpes) 

p 

ESTABLISH  STANDING  REQUEST  FOR  INFORMATION 

S 
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Table  A-3c 


Message  Code  List  To  Be  Learned  In  Response  To  Message  Title 
And  Type  Of  Action  In  the  Area  Of  G3  Functions 


No. 

Messaae  Title  And  Action 

Messaqe 

1 

Task  firgani zation-ADD 

UTOA 

2 

Task  Organization-CHANGE 

UTOC 

3 

Task  Organization-DELETE 

UTOD 

4 

Task  Organization-QUERY 

UTOO 

5 

Task  Organization-ESTABLISH  SRI 

UTOS 

6 

Task  Force-ADD 

UTFA 

7 

Task  Force-CHANGE 

UTFC 

8 

Task  Force-DELETE 

UTTD 

9 

Task  Force-QUERY 

UTFQ 

10 

Task  Force-ESTAELISH  SRI 

UTFS 

11 

Tactical  Disposi tions-QUERY 

UTDO 

12 

Tactical  Disposi tions-SPECIAL  PROCESSING 

UTDP 

13 

Tactical  Dispositions-ESTABLISI!  SRI 

UTDS 

14 

Unit  Disposition,  One  Coordinate-ADD 

UDCA 

15 

Unit  Disposition,  One  Coordi nate-CHANGE 

UDCC 

16 

Unit  Disposition,  One  Coordi nate-DELETE 

UDCD 

17 

Unit  Disposition,  One  Coordinate-CUERY 

UDCO 

18 

Unit  Disposition,  One  Coordinate-ESTABLISH  SRI 

UDCS 

19 

Unit  Disposition,  One  Unit-ADD 

UDUA 

20 

Unit  Disposition,  One  Unit-CHANGE 

UDUC 

21 

Unit  Disposition,  One  Unit-DELETE 

UDim 

22 

Unit  Disposition,  One  Unit-OUERY 

uduo 

23 

Unit  Disposition,  One  Uni t-ESTABLISH  SRI 

UDIJS 

24 

Unit  Disposition,  General -ADD 

UDGA 

25 

Unit  D'isposi tion.  General -CHANGE 

UDGC 

26 

Unit  Disposition,  General-DELETE 

UDGD 

27 

Unit  Disposition,  General -Ol'ERY 

UDGO 

28 

Unit  Disposition,  General-ESTABLISH  SRI 

UDGS 

29 

Air  Control  Measure- ADD 

ACMA 

30 

Air  Control  Measure-OIANGE 

ACMC 

31 

Air  Control  Measure-DELETE 

ACMD 

32 

Air  Control  Measure-OUERY 

ACMO 

33 

Air  Control  Measure-ESTABLISH  SRI 

ACMS 
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Table  A-3c,  Continued 


No. 

Message  Title  and  Action 

Message  Code 

34 

Operations  Journal -ADD 

UOJA 

35 

Operations  Journal -DELETE 

UOJD 

36 

Operations  Journal-QUERY 

UOJQ 

37 

Operations  Journal -SPECIAL  PROCESSING 

UOJP 

38 

Situation  Report-CHANGE 

IJSIC 

39 

Situation  Report-SPECIAL  PROCESSING 

USIP 

40 

Pending  Change-ADD 

UPCA 

41 

Pending  Change-CHANGE 

UPCC 

42 

Pending  Change-DELETE 

UPCD 

43 

Pending  Change-QUERY 

UPCO 

44 

Spot  Report-ADD 

USRA 

45 

Spot  Report-QUERY 

USRO 

46 

Aircraft  Avail able-ADD 

AAV  A 

47 

Aircraft  Avail able-CHANGE 

AAVC 

48 

Aircraft  Avai lable-DELETE 

AAVD 

49 

Aircraft  Available-QUERY 

AAVQ 

50 

Aircraft  Avai lable-ESTABLISH  SRI 

AAVS 

51 

Relay-ADD 

RELA 

52 

Named  Area  of  Interest-ADD 

NAIA 

53 

Named  Area  of  Interest-CEANGE 

NAIC 

54 

Named  Area  of  I nterest-DELETE 

NAID 

55 

Named  Area  of  Interest-OUERY 

NAIO 

56 

Standing  Request  for  Information  Eile-DELETE 

SRID 

57 

Standing  Request  for  Information  File-OUERY 

SRIO 
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Tahle  A-4a 

Message  Titles  And  Their  Abbreviate  ns.  For  G3  Functions 


Message  Title 


No. 

Message 

Abbreviati 

1 

Task  Organization 

~W~ 

2 

Task  Force 

UB 

3 

Tactical  Dispositions 

IJD 

4 

Unit  Disposition,  One  Coordinate 

UE 

5 

Unit  Disposition,  One  Unit 

UF 

6 

Unit  Disposition,  General 

UG 

7 

Air  Control  Measure 

UH 

8 

Operations  Journal 

UL 

9 

Situation  Report 

UM 

10 

Pending  Change 

UN 

11 

Spot  Report 

US 

12 

Aircraft  Available 

FA 

13 

Relay 

AA 

14 

Named  Area  of  Interest 

AA 

15 

Standing  Request  for  Information  File 

AA 

Table  A-4b 

Types  of  Actions  And  The  Numerical  Code  For  Each 


Action  Desired  Recording  Message Action  Code 

7\DD  (to  data  basel  * T 
CHANGE  (the  data  base)  2 
DELETE  (from  data  base)  3 
nUERY  (the  data  base)  6 
SPECIAL  PROCESSING  (of  some  message  tvnes)  5 
ESTABLISH  STANDING  REQUEST  TOR  INFORMATION  6 


rn  Only  an  Action  Code  of  0 may  be  used  with  the 
Relay  Message. 

(2)  Use  an  Action  Code  of  8 to  delete  an  SRI  File 
message. 

(3)  Use  an  Action  Code  of  9 to  query  the  SRI  File. 


Table  A-4c 


Message  Code  List  To  Be  Learned  In  Response  To  Message  Title 
And  Type  Of  Action  In  The  Area  Of  G3  Functions 


No. 

Message  Title  And  Action 

Messaqe  Code 

1 

Task  Organization-ADD 

“DAT 

2 

Task  Organization-CHANGE 

1IA  2 

3 

Task  Organization-DELETE 

UA  3 

4 

Task  Organization-QUERY 

IJA  4 

5 

Task  Organization-ESTAELISH  SRI 

UA  6 

6 

Task  Force-ADD 

UP  1 

7 

Task  Force-CHANGE 

UB  2 

8 

Task  Force-DELETE 

UB  3 

9 

Task  Force-QUERY 

UB  4 

10 

Task  Force-ESTABLISH  SRI 

UB  6 

11 

Tactical  Dispositions-OUERY 

UD  4 

12 

Tactical  Dispositions-SPECIAL  PROCESSING 

UD  5 

13 

Tactical  Dispositions-ESTABLISH  SRI 

UD  6 

14 

Unit  Disposition,  One  Coordinate-ADD 

UE  1 

15 

Unit  Disposition,  One  Coordi nate-CMANGE 

UE  2 

16 

Unit  Disposition,  One  Coordinate-DELETE 

UE  3 

17 

Unit  Disposition,  Ore  Coordi nate-QUERY 

UE  4 

18 

Unit  Disposition,  One  Coordi nate-FSTABLISH  SRI 

UE  6 

19 

Unit  Disposition,  One  Unit-ADD 

UF  1 

20 

Unit  Disposition,  One  Unit-CHANGE 

UF  2 

21 

Unit  Disposition,  One  'Jnit-DELETE 

UF  3 

22 

Unit  Disposition,  One  Unit-DIJERY 

UF  4 

23 

Unit  Disposition,  One  L'ni t-ESTACLISH  SRI 

UF  fi 

24 

Unit  Disposition,  General -ADD 

UG  1 

25 

Unit  Disposition,  General -CHANGE 

UG  2 

26 

Unit  Disposition,  General -DELETE 

UG  3 

27 

Unit  Disposition,  General -OUERY 

UG  4 

28 

Unit  Disposition,  General -ESTABLISH  SRI 

UG  6 

29 

Air  Control  Measure-ADD 

UH  1 

30 

Air  Control  Measure-CHANGE 

UH  2 

31 

Air  Control  Measure-DELETE 

UH  3 

32 

Air  Control  Moasure-OUFRY 

UH  4 

33 

Air  Control  Measure-ESTABLISH  SRI 

UH  6 

34 

Operations  Journal -ADD 

UL  1 

35 

Operations  Journal -DELETE 

UL  3 

36 

Operations  JOurnal -QUERY 

UL  4 

37 

Operations  Journal -SPECIAL  PROCESSING 

UL  5 
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Table  A-4c,  Continued 


No. Message  Title  And  Action Message  Code 


38 

Situation  Report-CHANGE 

UM  2 

39 

Situation  Report-SPECIAL  PROCESSING 

UM  5 

40 

Pending  Change-ADD 

UM  1 

41 

Pending  Change-CHANGE 

UN  2 

42 

Pending  Change-DELETE 

UN  3 

43 

Pending  Change-OUERY 

UN  4 

44 

Spot  Report-ADD 

US  1 

45 

Spot  Report-OUERY 

US  4 

46 

Aircraft  Available-ADD 

FA  1 

47 

Aircraft  Avai 1 able-CHANGE 

FA  2 

48 

Aircraft  Avai lab! e-DELETE 

FA  3 

49 

Aircraft  Avai lable-OUERY 

FA  4 

50 

Aircraft  Avai lable-ESTABLISH  SRI 

FA  6 

51 

Relay-ADD 

AA  0 

52 

Named  Area  of  Interest-ADD 

AA  1 

53 

Named  Area  of  Interest-CHANGE 

AA  2 

54 

Named  Area  of  Interest^-DELETE 

AA  3 

55 

Named  Area  of  Interest-OUERv 

AA  4 

56 

Standing  Request  for  Information  Fil e-DELETE 

AA  8 

57 

Standing  Request  for  Information  File-OUERY 

AA  9 
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APPENDIX  B 


STATISTICAL  ANALYSIS  TABLES 

This  appendix  contains  tabular  presentations  of  the  results  of  the 
several  analyses  of  variance  performed  on  the  data.  The  ordering  of  the 
tables  follows  the  order  in  which  the  dependent  variables  are  presented 
for  a given  independent  variable;  i.e.,  number  of  passes  to  reach  the 
learning  criterion,  percent  errors  made,  and  time  to  input  a message 
reference  code  following  presentation  of  a message  title  and  action  name. 

Because  the  number  of  subjects  in  the  six  groups  was  not  equal 
(n  = 10,  10,  10,  13,  9,  8)  calculation  of  sums  of  squares  by  usual 
procedures  produced  some  negative  interaction  sums  of  squares  and  mean 
squares.  The  method  recommended  by  Walker  and  Lev*  on  pp.  381-382  to 
handle  unequal  frequencies  was  followed  in  all  of  the  2x2  ANOVAS  performed. 


♦Walker,  H.M.  & Lev,  J.  Statistical  Inference.  Kenrv  Holt  and  Cnmnanv 
New  York:  1953. 


PRECEDING  PACK  BLaMC-NOT  FILMED 


Table  B-l 


ANOVA  OF  NUMBER  OF  PASSES  ON  G2  LIST 
FOR  LL#  vs.  LLLL  CODES  AND  OFFICERS  vs.  EM 


aError  Mean  Square  = 316.4923+>  39  x 0.0942  = 0.7645 
where  0.0942  = mean  reciprocal  of  sample  sizes  and 
316.4923  = Error  S.S.  from  usual  ANOVA. 


Prob. 

p*  .001 

.01*  p * .05 
p>  .50 


i 


Table  B-2 

ANOVA  OF  NUMBER  OF  PASSES  BY  OFFICERS 
FOR  IU  vs.  LLLL  CODES  AND  G2  vs.  G3  LISTS 


Source 
Code 
List 
C x L 
Error 


S S 
5.8153 
2.2907 
.3838 
a 


2.2907 
0.3838 
0 . 3 1 21 a 


Prob. 

TT750T 

p = .01 

.25*  p*  .30 


aError  Mean  Square  = 108.7827+  36  x 0.1033  = 0.3121 
where  0.1033  = mean  reciprocal  of  sample  sizes,  and 
108.7827  = Error  S.S.  from  usual  ANOVA. 
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Table  B-3 


ANOVA  OF  PERCENT  ERROR  ON  C?  LIST  FOR  LL#  vs.  LLLL 
COOES  AND  OFFICERS  vs.  EM 


Source 

S.S. 

df 

M.S.  F 

Prob. 

Code 

179.1583 

1 

179.1 533  26.94 

VQQiTllH 

Rank 

26.6773 

1 

C x R 

6.0269 

1 

6.0269  0.91 

Error 

a 

39 

6. 6495a — - 

aError  Mean  Square  = 70.5891  -f  39  x 0.0942  = 6.6495, 
where  0.0942  = mean  reciprocal  of  sample  sizes,  and 
70.5891  = Error  S.S.  from  usual  ANOVA. 

Source 

ANOVA  OF  PERCENT 
CODES 

S.S.  df 

Table  B-4 

ERROR  BY  OFFICFRS  FOR  LL« 
AND  G2  vs.  G3  LISTS 

M.S.  F 

vs.  LLLL 

Prob. 

Code 

5576W 

r 

95.6484 

29.60 

p<;  .001 

List 

26.5226 

i 

26.5226 

8.21 

P<  -01 

C x L 

1.3224 

i 

1.3224 

0.41 

p?  . 50 

Error 

a 

36 

3. 2309a 

— 

— 

aError  Mean  Square  = 
where  0.1033  = Mean 
and  31.2768  = Error 


31.2768  f 36  x 0.1033  = 3.2309, 
reciprocal  of  sample  sizes, 

S.S.  from  usual  ANOVA. 
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Table  B-5 


ANQVA  OF  TIME  TO  INPUT  A MESSAGE  REFERENCE  CODE 
FOR  G2  LIST  FOR  LL#  vs.  LLLL  CODE  AND  OFFICERS  vs.  EM 


Source 


S.S. 


Code 
Rank 
C x R 
Error 


.0037 

.0234 


df 

I 

1 

39 


M.S. 

row 


F 


.0037 

.0234 

,2052a 


0.02 

0.11 


Prob. 
p > . 5(T 
p>  .50 
p>  .50 


aError  Mean  Square  = 84.9721 -r-  39  x 0.0942  = 0.2052, 
where  0.0942  = mean  reciprocal  of  sample  sizes, 
and  84.9721  = Error  S.S.  form  usual  ANOVA. 


Table  B-6 


ANOVA  OF 

TIME  TO 

INPUT  A MESSAGE 

REFERENCE 

CODE 

BY  OFFICERS 

FOR  IU 

vs.  LLLL  CODES 

AMD  G2  vs. 

G3  LISTS 

Source 

S.S. 

df 

M.S. 

F 

Prob. 

Code 

r 

0.01 

p > . 50 

List 

.6942 

1 

.6942 

3.26 

p = .08 

C x L 

.0629 

1 

.0629 

0.30 

p>.50 

Error 

a 

36 

. 21 30a 

- - - 

- - “ 

aError  Mean  Square  = 74.2774 -4*  36  x 0.1033  = 0.2130, 
where  0.1033  = mean  reciprocal  of  sample  sizes, 
and  74.2774  = Error  S.S.  from  usual  ANQVA. 


Table  B-7 


t-TESTS  OF  DIFFERENCE  BETWEEN  MEANS  OF  CORRELATED  MEASURES 
OF  PERCENT  ERRORS  FOR  1st  and  2nd  CHARACTER  OF  THE  LLif  CODE 
(Officer  Data  Only) 


G2  List 

G3  List 

Overall 

1st  Letter 

4.55% 

3.49% 

4.05% 

2nd  Letter 

13.34% 

10.99% 

12.23% 

Difference 

8.79% 

7.50% 

8.18% 

SD 

1.3924% 

1.1404% 

0.8983 

t 

6.31 

6.576 

9.10 

Probabi 1 i tv* 

< .001 

4 .001 

< .001 

*two-tai led 


Table  B-8 

t-TESTS  OF  DIFFERENCES  BETWEEN  MEANS  OF  CORRELATED  MEASURES  OF 
PERCENT  ERRORS  FOR  1st,  2nd  and  3rd  CHARACTERS  OF  THE  LLLL  CODE 

(Officer  Data  Only) 


G2  List 

G3  List 

1st  Letter 

3.25% 

3.81% 

2nd  Letter 

5.35% 

2.10% 

Difference 

2.092% 

-1.7125% 

SD 

0.6180% 

0.4991% 

t 

3.39 

3.43 

d.f. 

12 

7 

Probability* 

< .01 

<.  .02 

2nd  Letter 

5.35% 

2.10% 

3rd  Letter 

7.13% 

3.65% 

Difference 

1.7846% 

1 . 55% 

SD 

0.6299% 

0.2449% 

t 

2.83 

6.33 

d.f. 

12 

7 

Probabi 1 i ty* 

1 

1 

C\J 

o 

V 

< .001 

1st  Letter 

3.25% 

3.81% 

3rd  Letter 

7.13% 

3.65% 

Difference 

3.877% 

-0.1625% 

SD 

0.9809% 

0.5318% 

t 

3.95 

-0.31 

d.f. 

12 

7 

d.f. 

Probability* 


< .005 


>.50 


Table  B-9 

t-TESTS  OF  DIFFERENCES  BETWEEN  MEANS  OF  CORRELATED  MEASURES 


OF  PERCENT  ERRORS 

FOR  MESSAGE  TITLE 
(Officer  Data 

CODES  AND 
Only) 

ACTION  NAME  CODES 

LL#  CODE 

G2  List 

G3  List 

Overal 1 

Title  Code 

8.96% 

7.23% 

8.14% 

Action  Code  (f*) 

13.11% 

6.24% 

9.86% 

Difference 

-4.15% 

0.9888% 

-1.7158% 

SD 

1.0772% 

0.7873% 

0.8953% 

d.f . 

9 

8 

18 

t 

-3.85 

1.25 

-1.92 

Probabi 1 ity* 

<.01 

* .26 

= .074 

*two-tai led 

LLLL  CODE 

G2  List 

G3  List 

Overall 

Title  Code 

5.25% 

3.20% 

4.47% 

Action  Code  (L) 

4.22% 

1.60% 

3.22% 

Difference 

1.0385% 

1.60% 

1 . 2524% 

S- 

D 

0.5763% 

0.7300% 

0.4451% 

d.f. 

12 

7 

20 

t 

1.80 

2.19 

2.81 

Probability* 

= .098 

= .068 

- .011 

Table  B-10 


t-TESTS  OF  DIFFERENCES  IN  MEAN  PERCENT  ERRORS  FOR  NUMERIC  VERSUS 
ALPHABETIC  ACTION  CODES. 

(Officer  Data  Only) 


G2  List 

G3  List 

Overal 1 

Mean  (#) 

13.11% 

6.24% 

9.86% 

Mean  (L) 

4.22% 

1.60% 

3.22% 

Variance  (#) 

25.2277 

6.4903 

27.9026 

n(#) 

10 

9 

19 

Variance  (L) 

6.3664 

1.7114 

6.1126 

n (L) 

13 

8 

21 

S.D.  (diff.  of  means) 

1.7356% 

0.9670% 

1.3265% 

d.f . 

13* 

13* 

26* 

t 

5.12 

4.80 

5.01 

Probability 

< .001 

<.001 

<.001 

I 

♦Computed  by  Walker  and  Lev's  formula  (7.26)  on  page  158. 


APPENDIX  C 


ANALYSIS  OF  ERRORS  BY  CHARACTER  POSITION 

Errors  By  Character  Position 

To  gain  a better  understanding  of  the  sources  of  error  and  difficulty 
in  learning  the  LL#  and  LLLL  codes,  a count  of  errors  by  character  position 
was  made  for  each  officer  within  each  of  the  four  combinations  of  list  type 
(G2  and  G3)  and  code  type.  These  error  counts  were  then  divided  by 
each  officer's  number  of  chances  to  err.  Summing  across  officers  by  char- 
acter position  and,  for  other  comparisons,  across  the  message  title 
character  positions  as  well,  a measure  of  the  mean  percent  error  by  char- 
acter position  (or  by  set  of  character  positions)  was  calculated.  The 
results  reveal  some  differences  not  capable  of  being  shown  by  the  pre- 
viously discussed  dependent  variable,  percent  of  rosnonses  containing 
at  least  one  error. 

Effect  of  the  Second  Letter  of  the  LL#  Code 

Error  data  regarding  the  second  letter  of  the  LL-  codes  give  strong 
support  to  the  prediction  that  its  arbitrariness  and  the  gaps  in  its 
alphabetic  sequence  would  be  a major  source  of  error  in  learning  the 
LL#  codes.  Computed  over  the  G2  and  G3  lists  (Table  C-l),  the  mean 
percent  error  for  the  second  letter  (12.23")  was  three  tines  that  for  the 
first  letter  (4.05").  The  ratio  is  slightlv  less  than  3 to  1 for  the  GT 
list  and  slightly  more  than  3 to  1 for  the  G3  list.  Three  t-tests  of 
these  differences  between  means  (13.34%  vs.  4.55%,  10.99%  vs.  3.49%,  and 
12.23%  vs.  4.05%)  each  indicated  statistical  significance  (see  Table  B-7) 
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Table  C-l 


SUMMARY  OF  MEAN  ERROR  PERCENTAGES  BY  CHARACTER  POSITION 
(Officer  Data  Only) 


List 

Code 

Message  Title 

Code  Characters 

1st  2nd  3rd 

Title 

Code 
Overal 1 

Action 

Code 

Character 

Message 

Reference 

Code 

G2 

LL# 

4.55 

13.34 

NA 

8.96 

13.11 

10.34 

G2 

LLLL 

3.25 

5.35 

7.13 

5.25 

4.22 

5.00 

G3 

LL# 

3.49 

10.99 

NA 

7.23 

6.24 

6.91 

G3 

LLLL 

3.81 

2.10 

3.65 

3.20 

1.60 

2.80 

— 

LL# 

4.05 

12.23 

NA 

8.14 

9.86 

8.72 

— 

LLLL 

3.47 

4.11 

5.80 

4.47 

3.22 

4.16 
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with  the  probability  of  obtaining  at  least  the  observed  differences  being 
less  than  .001  under  an  hypothesis  of  no  difference.  Examination  of 
Tables  C-l  and  C-2  shows  that  the  mean  percent  errors  made  on  the  second 
letter  exceed  those  for  the  first  letter  for  16  of  the  17  message  title 
codes  other  than  the  "AA"  codes.  The  "AA"  codes  are  especially  compli- 
cated, the  result  being  that  similar  error  rates  were  associated  with 
each  of  the  two  letters.  The  remaining  exception  is  the  ''FA"  message 
code  where  6.2%  errors  were  associated  with  the  "F"  and  4.3  errors  with 
the  "A"  (Table  C-3). 

It  seems  clear  that  the  second  letter  of  the  LL#  code  was  a major 
source  of  error  because  its  characters  had  very  little  mnemonic  value 
and  was  even  difficult  to  learn  as  a sequence  of  the  alphabet  because 
of  gaps.  It  also  appears  that  other  factors,  such  as  relative  fre- 
quency and  recency  of  occurrence,  influence  the  learning  of  the  code 
characters  so  that  in  a certain  minority  of  cases  the  first  character 
is  as  hard  or  harder  to  learn  than  the  second  character. 

Error  Relations  in  the  Message  Portion  of  the  LLLL  Code 

Neither  the  second  nor  third  character  of  the  LLLL  code  has  three 
times  the  error  percentage  of  the  first  character,  as  was  noted  for  the 
LL#  code.  Yet,  in  the  case  of  the  G2  list,  higher  error  percentages 
occur  for  the  second  character  than  for  the  first  character,  and  still 
higher  error  percentages  occur  for  the  third  than  for  the  second  char- 
acter (3.25",  then  5.35",  then  7.13";  see  Table  C-l).  From  data  in  Table 
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Table  C-2 


Message 

HUe..f,od£. 


A A 

10.8  ' T1.0 


MEAN  PERCENT  ERRORS  BY  CHARACTER  POSITION 
G2  List,  LL#  Code,  10  Officers 

Action  Code 


E 

A 

1 

★ 

3 

4 

4.7 

6.6 

10.4 

14. ’4 

4.3 

E 

c 

1 

2 

3 

4 

6 

477“ 

"7077 

"7775 

9.9 

8.6 

4.0 

49.5 

E 

D 

1 

2 

3 

5 

470“ 

TO 

4.8 

6.4 

6.1 

45.4 

E 

E 

1 

2 

3 

4 

1.4 

11.0 

3.1 

3.7 

3.2 

2.8 

E 

G 

1 

2 

4 

5.4 

' 22.6 

4.0 

2.8 

13.0 

A 

A 

0 

13.6 

5.9 

20.1 

A 

A 

1 

2 

3 

4 

2.4 

1.8 

9.6 

13.6 

8.9 

5.0 

1st  2nd  3rd 
4.6  13.3  1 3 . 1 


1 

"O' 


2 

77 T 


3 

757T 


4 

“575" 


45.4 


49.5 


0 

ToTT 


8 


4571  387  6 
8 9 


4577  3876 


*Gaps  such  as  this  occur  because  not  all  action  codes  are  applicable  to  each 
message  category. 


Table  C-3 


MEAN  PERCENT  ERRORS  BY  CHARACTER  POSITION 
(G3  List,  LL # Code,  9 Officers) 

Message 

Title  Code  Action  Code 


r 


A-4,  one  can  infer  that  the  higher  error  rates  for  the  second  and  third 
characters  are  mainly  the  result  of  troubles  in  learning  EU[  for  (Enemy) 
Intelligence  Summary,  EWF  for  (Enemy)  Intelligence  Work  File,  and  EDX_ 
for  Enemy  Situation  Data  Case  Index.  The  third  character  incurred  higher 
error  rates  than  the  second  character  primarily  because  learners  persisted 
in  responding  "ElSVather  than  "EIN"  to  the  title  (Enemy)  Intelligence 
Sumnary.  The  difference  in  mean  percentage  errors  between  the  first  and 
second  characters  of  the  G2  list,  LLLL  code,  was  statistically  significant 
(p  < .01 ) and  so  was  the  difference  between  the  second  and  third  characters 
(p  <.02;  see  Table  B-8) . 

The  pattern  of  errors  among  the  first  three  characters  on  the  G3 
list  (Table  C-l ) was  different  from  that  of  the  G2  list.  For  the  G3 
list,  the  mean  percent  errors  for  the  second  character  (2.10»)  was  less 
than  that  associated  with  either  the  first  character  (3.81")  or  the  third 
character  (3.65").  While  these  differences  were  statistically  significant 
(p<  .02  and  p<  .001),  the  difference  between  the  first  and  third  character's 
error  rates  was  not  statistically  significant.  A likely  reason  for  the 
lower  error  rates  for  the  second  character  is  that  many  of  the  G3  titles 
contained  just  two  words.  In  these  cases  subjects  appeared  to  learn 
rather  quickly  that  they  should  enter  a "U"  for  the  first  character  and 
then  enter  the  acronym  of  the  first  title  word  as  the  second  character 
of  the  code. 

Message  Title  Errors  Versus  Action  Name  Errors 

For  just  one  of  the  four  combinations  of  list  and  code  type  was  the 
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Table  C-4 


r 

V 


MEAN  PERCENT  ERRORS  BY  CHARACTER  POSITION 
(G2  List,  LLLL  Code,  13  Officers) 

Message  Title  Code  Action  Code 


— — — ’ ~ ^ ~ - 

mean  error  percentage  for  the  action  code  greater  than  that  for  the  message 
title  code.  The  numeric  action  codes  of  the  G2  list  incurred  13.11% 
errors  while  the  message  title  characters  of  this  list  incurred  8.96% 
errors.  The  difference  between  these  means  was  statistically  significant 
(p<.01;  see  Table  B-9).  For  the  G3  list,  LL#  code,  the  action  portion 
had  a lower  error  rate  (6.24%)  than  the  title  portion  (7.23"')  but  the 
difference  was  not  significant  (p  = .26). 

For  the  LLLL  Code,  a statistically  significant  difference  was 
obtained  only  when  error  percentages  were  computed  over  both  G2  and  G3 
lists.  For  this  comparison  the  title  portion  incurred  4.47%  errors  while 
the  alphabetic  action  code  portion  incurred  3.22%  errors  (p  = .011). 

Overall,  the  action  code  letter  was  input  with  3.22%  errors  while 

Ithe  action  code  as  a number  was  input  with  9.86%  errors  (a  statistically 
significant  difference;  p < . 001  from  Table  B-10).  Since  the  error  rate 
for  the  message  title  portion  dropped  from  8.14%  for  the  LL#  codes  to 
4.47%  for  the  LLLL  codes,  we  find  the  new  LLLL  code  has  been  more  helpful 
in  reducing  action  code  error  rates  than  in  reducing  message  title  cede 
error  rates.  Evidently  the  numeric  action  codes  with  their  exceptions  were 
somewhat  more  difficult  than  expected.  On  the  other  hand,  the  new  LLLL 
title  codes  were  somewhat  less  of  an  improvement  than  had  been  hoped  for. 

Action  Code  Problems 

The  LL#  code  contains  nine  different  action  codes,  all  numbers, 
compared  to  only  six  different  letters  serving  as  the  action  codes  in  the 
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LLLL  code  system.  The  extra  three  action  codes  (0,  8,  9)  apply  to  action 
names  that  most  often  require  a response  of  1 , 3,  and  4.  Apparently  a 
system  designer  felt  that  the  fact  that  the  three  message  categories 
involved  (Relay,  Named  Area  of  Interest,  and  SRI  File)  were  common  to  both 
G2  and  G3,  warranted  giving  them  all  the  same  letter-letter  (AA)  designation. 
Then,  to  provide  unique  action  code  numbers,  for  the  computer  program,  he  was 
forced  to  use  a "0"  to  distinguish  Relay-Add  from  Named  Area  of  Interest-Add; 
to  use  an  "8"  to  distinguish  Standing  Request  For  Information  File-Delete, 
from  Named  Area  of  Interest-Delete;  and  to  use  a "9"  to  distinguish  SRI  File- 
Query  from  Named  Area  of  Interest-Query.  The  high  error  rates  for  the 
0,  8,  9 action  code  numbers  show  clearly  that  the  use  of  the  common  AA 
designation  was  an  unnecessary  and  undesirable  complication  (See  Tables  C-2 
and  C-3) . 

For  the  LLLL  code  and  the  G2  list  (Table  C-4)  the  worst  action  code 
letter  is  the  "P"  standing  for  "Special  Processing."  This  should  be  con- 
sidered in  conjunction  with  the  "S"  that  stands  for  Establish  Standing 
Request  for  Information.  These  two  had  error  rates  of  23.8%  and  17.9% 
for  the  G2  list,  where  they  occurred  only  once  per  pass.  On  the  G3 
list,  (Table  C-S)  "P"  was  required  three  times  per  pass  and  "S"  was  required 
six  times  per  pass.  Since  their  error  rates  dropped  to  2.9%  and  1.3% 
overall  on  the  G3  list,  their  rarity  or  low  frequency  would  appear  to  be 
the  major  factor  producing  the  high  error  rates  on  the  G2  list.  G2 
learners  need  help;  G3  learners  apparently  need  little  help  if  any,  with 
the  "P"  and  "S."  Review  of  the  G2  learners'  wrong  responses  indicated 
that  many  were  responding  with  an  "S"  to  Special  Processing  and  with  an 
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"E"  to  Establish  SRI.  They  were  using  the  acronym  rule  that  most  often 
did  apply.  Unfortunately  for  their  learning's  success,  these  two  codes 
were  requiring  use  of  the  first  letter  of  the  second  word  of  the  action 
name  instead  of  the  first  letter  of  the  first  word.  The  reason  for  the 
original  assignment  of  "S"  to  Establish  SRI  was  that  it  seemed  appropriate 
or  naturally  associated  with  SRI.  But  this  assignment  ignored  the  need  to 
use  the  verb  "establish"  to  describe/explain  the  action  involved,  and  it 
took  insufficient  account  of  the  error-producing  consequences  of  letting 
"P"  stand  for  Special  Processing  now  that  the  "S"  was  pre-empted.  It  is 
thought  that  changing  the  code  letter  for  Special  Processing  to  "S"  and 
the  code  letter  for  Establish  SRI  to  "E"  would  aid  the  learning  of  the 
G2  list. 

"Establish  SRI"  is  quite  clear  in  its  meaning  of  the  action.  "Initiate 
SRI"  seems  a bit  more  precise,  but  to  change  the  name  of  the  action  to  this 
(with  a code  letter  of  "I")  seems  unlikely  to  reduce  error  rates  below  that 
which  the  use  of  "E"  would  produce.  The  action  name  of  "Start  SRI,"  using 
a code  of  "S,"  would  allow  retention  of  the  "S"  code  with  perhaps  not  too 
significant  a loss  in  conveying  the  meaning  of  the  action  to  the  learner. 

Individual  Message  Title  Code  Problems 

The  letters  EIN  were  the  code  response  required  to  the  message  category 
titles  displayed  as  (Enemy)  Intelligence  Summary.  The  "N"  has  a 13.2% 
error  rate,  and  the  common  error  was  to  give  "S"  as  the  third  letter.  It 
seems  obvious  that,  given  this  display  of  the  message  title,  EIS  should  have 
been  the  required  code.  The  situation  here  has  been  complicated  by  the 


addition  of  (Enemy)  in  the  display  of  the  message  title;  this  word  does 
not  appear  in  the  title  printed  on  message  format  sheets.  It  was  displayed 
parenthetically  to  justify  the  use  of  "E"  as  the  first  letter  of  the  code 
and  to  create  three  words  in  the  title  thus  allowing  straightforward 
application  of  the  three  letter  acronym  rule  for  this  title.  If  all  message 
reference  codes  must  have  four  letters,  "EIS"  coupled  with  the  use  of 
"(Enemy)  Intelligence  Summary"  as  a title,  is  judged  to  be  a combination 
that  would  produce  a lower  error  rate  than  "EIN"  did. 

In  the  next  case,  however,  the  addition  of  "(Enemy)"  to  "Intelligence 
Work  File"  disallowed  simple  application  of  the  three  letter  acronym  rule. 
The  "W"  and  "F"  each  had  a 10"  error  rate.  Subjects  were  charged  with  an 
error  for  inserting  the  letter  "I"  and  were  unfortunately  required  to 
leave  this  letter  out  of  the  code.  The  title  displayed  should  have  been 
simply  "Intelligence  Work  File"  and  the  code  for  it  should  be  "IWF"  plus 
the  appropriate  action  code. 

The  next  G2  problem  is  the  five-word  title  "Enemy  Situation  Data  Base 
Index"  (code  is  EDX)  where  use  of  a three  letter  title  code  requires  the 
subject  to  learn  which  three  of  the  five  words  contain  the  code  letters. 

The  "E"  is  the  best  first  letter  code;  while  the  "D"  was  assigned  without 
expectation  that  it  would  be  more  readily  learned  than  "S"  or  "B,"  the 
two  remaining  words  '■  first  letters.  "X"  stands  out  as  a cue  for  index, 
but  creates  an  unnecessary  exceptional  case,  for  it's  not  the  first  letter 
of  a word.  The  error  responses  were  usually  "ESD. " Again,  this  shows 
that  on  these  occasions  the  learners  were  trying  to  follow  a simple 
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acronym  rule.  Unfortunately,  the  existence  of  another  message  category 
title  "Enemy  Situation  Data"  pre-empts  the  use  of  "ESD"  as  a code.  A 
change  in  title  seems  much  needed  in  this  case  in  order  to  permit  simple 
application  of  the  three  letter  acronym  procedure  for  generating  or  deter- 
mining the  title  code.  "Situation  Data  File"  (Code  to  be  "SDF")  and 
"Situation  Data  Index"  (Code  to  be  "SDI")  are  candidate  title  reductions. 
It  is  thought  that  either  (the  latter  is  somewhat  preferred)  would  reduce 
error  rates  below  what  are  produced  by  the  present  combination  of  "EDX" 
and  a five  word  title. 

"Relay-Add"  is  used  by  both  G2  and  G3  operations.  The  error  rate 
for  the  three  code  letters  "REL"  was  virtually  the  same  (12.4%,  12.2%,  and 
12.2%)  for  each  letter  for  the  G2  list  and  was  the  same  (15%)  for  each 
letter  when  being  learned  by  subjects  assigned  to  the  G3  list.  Most  of 
the  errors  regarding  the  "R"  were  of  a premature  commission  nature  - 
subjects  input  the  R before  the  display  had  stopped  "rolling  up."  It  is 
apparently  coincidental  that  the  error  rates  for  the  R match  those  for 
the  other  two  letters.  Given  a one-word  message  title  and  a three-letter 
code  input  requirement,  "REL"  seems  fairly  obvious.  Combined  with  "A," 
the  only  action  code  permitted,  produces  "RELA"  which  seems  inspired  or 
at  least  clever.  To  adapt  it  to  a three  word  and  three  letter  acronym 
scheme,  the  cumbersome  title  "Common  Relay  Message"  and  associated  code 
"CRM"  is  a possibility. 

Only  one  other  character  on  the  G3  list  stands  out  with  a high 
error  rate.  Situation  Report  is  encoded  as  "USI,"  and  the  error  rate 
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for  the  "I"  (Table  C-5)  is  27.2%.  The  bulk  of  the  "I"  errors  come  about 
because  the  subjects  are  inputting  "USR."  "USR"  is  the  more  appropriate 
code,  but  it  is  already  the  code  for  "Spot  Report."  This  duplication  of 
first  letters  ("S"  and  "R")  for  two  different  titles  is  not  simple  re- 
solvable while  staying  within  the  acronym  procedure.  It  seems  necessary 
to  change  one  of  the  two  titles  to  enable  formulation  of  a different 
acronymic  code.  The  short  form  "SITREP"  could  be  used,  and  as  with  "Relay" 
the  first  three  letters  "SIT"  employed  as  a (nonacronvmic)  code.  However, 
the  dropping  of  V from  the  code  should  cause  error  rates  to  stay  fairly 
high,  for  "U"  (standing  for  "unit")  is  frequently  used  in  the  G3  list  as 
a filler  letter  when  the  message  title  contains  only  two  v/ords. 
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Table  C-5 

MEAN  PERCENT  ERRORS  BY  CHARACTER  POSITION 
(G3  List,  LLLL  Code,  8 Officers) 


Messaqe  Title  Code  Action  Code 


Table  D-2 


SUBJECT 

1_ 

ERRORS  BY 

2 3 

PASS 

4 

: EM,  G2 

PASS 

5 6 

LIST, 

7 

LLLL 

8 

CODE 

9 

10 

E2 

Ta 

4 

4 

2 

1 

0 

11 

E4 

RC 

14 

11 

7 

7 

6 

5 

1 

2 

5 

0 

58 

E4 

CR 

5 

7 

3 

2 

3 

1 

0 

21 

E4 

Na 

4 

1 

0 

5 

E4 

Ma 

1 

0 

1 

E6 

WT 

6 

4 

0 

10 

E5 

WB 

10 

13 

4 

2 

5 

3 

2 

0 

39 

E5 

Pa 

8 

7 

3 

2 

0 

20 

E7 

Ch 

7 

7 

2 

0 

16 

CO 

Lul 

Fi 

15 

6 

2 

2 

0 

25 

i 
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Table  D-3 


ERRORS  BY  PASS:  OFFICERS,  G2  LIST,  LU  CODE 

PASS 


SUBJECT 

I 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

£ E 

04 

Go 

10 

20 

16 

24 

15 

11 

12 

8 

3 

2 

3 

0 

124 

04 

Sc 

5 

13 

6 

6 

1 

4 

2 

1 

0 

38 

04 

Gi 

7 

8 

3 

1 

1 

1 

0 

21 

03 

Wa 

6 

11 

4 

3 

0 

24 

03 

Ot 

10 

16 

9 

5 

2 

0 

42 

03 

Sp 

14 

11 

5 

0 

30 

03 

Ze 

4 

7 

3 

3 

0 

17 

03 

Qu 

6 

5 

0 

11 

05 

St 

12 

12 

9 

7 

3 

2 

1 

0 

46 

02 

Br 

10 

21 

7 

4 

4 

4 

4 

1 

0 

55 
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Table  D-4 


ERRORS  BY  PASS:  OFFICERS,  G2  LIST,  LLLL  CODE 
PASS 


SUBJECT 

1 

2 

3 

4 

5 

6 

04 

DS 

4 

7 

0 

11 

04 

Pr 

5 

12 

6 

5 

2 

0 

30 

04 

Ke 

1 

0 

1 

04 

WB 

5 

3 

1 

0 

9 

04 

RG 

4 

2 

0 

6 

04 

Hi 

7 

7 

0 

14 

04 

Mu 

4 

4 

0 

8 

04 

JG 

6 

5 

3 

1 

3 

0 

18 

02 

Bj 

3 

1 

0 

4 

03 

Lo 

4 

3 

1 

0 

8 

03 

Re 

3 

1 

2 

0 

6 

03 

Mi 

9 

5 

1 

0 

15 

04 

NP 

6 

3 

6 

1 

0 

16 

58 
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Table  D-5 


ERRORS  BY  PASS:  OFFICERS,  G3  LIST,  LL#  CODE 


SUBJECT 

I 

2 

3 

4 

5 

6 

7 

_1L 

04  La 

10 

23 

5 

2 

0 

40 

04  St 

17 

16 

3 

0 

36 

04  Pa 

12 

9 

2 

0 

23 

04  Su 

5 

19 

5 

0 

29 

04  SI 

11 

15 

0 

26 

04  Be 

13 

6 

0 

19 

03  Mu 

20 

25 

5 

4 

1 

0 

55 

04  Br 

18 

35 

13 

11 

11 

2 

0 

90 

05  Ch 

13 

33 

14 

3 

1 

C 

64 
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Table  D-6 


SUBJECT 
04  A1 

04  Ma 
03  Cu 
03  La 
03  Le 

05  Su 
03  Yo 
03  Me 
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Table  D-8 


Table  D-10 


DISTRIBUTION 


ARI  Distribution  List 


r 


4 

2 

2 

2 

2 

2 


OASD  (M&RA) 

HQDA  (DAMI  CSZ) 

HQDA  (DAPE  PBR 
HQDA  (DAMAAR) 

HQDA  (DAPE  HR6  PO) 

HQDA  (SGRO  ID\ 

HQDA  (DAMI  DOT  C) 

HQDA  (DAPC  PMZ  A) 

HQDA  IDACH  PPZ  A) 

HQDA  (DAPE  HRE) 

HQDA  (DAPF  MPO  C> 

HQDA  (DAPE  DW) 

HQDA  (DAPE  HRL) 

HQDA  (DAPE  CPS) 

HQDA  (DAFD  MFA) 

HQDA  (DARD  ARS  P) 

HODA  (DAPC  PAS  A) 

HQDA  (DUSA  OR) 

HQDA  (DAMO  RQR) 

HQDA  (DASG) 

HQDA  (OAIO-Pl) 

Chief.  Consult  Div  (DA  OTSG),  Adelphi,  MD 
Mil  Asst  Hum  Res.  ODDR&E.  OAD  (E&LS) 

HQ  USARAL.  APO  Seattle.  ATTN:  ARAGP  R 

HQ  F.rst  Army.  ATTN:  AFKA  OI  TI 

HQ  Fifth  Army,  Ft  Sam  Houston 

Dir.  Army  Stf  Stud.es  Ofc,  ATTN:  OAVCSA  (DSP) 

Ofc  Chief  of  Stf.  Studies  Ofc 

DCSPER.ATTN  CPS/OCP 


The  Army  Lib,  Pentagon,  ATTN:  RSB  Chief 
The  Army  Lib.  Pentagon,  ATTN:  ANRAL 
Ofc.  Asst  Sect  of  the  Army  (R&D) 

Tech  Support  Ofc,  OJCS 

USASA.  Arlington.  ATTN  IARD  T 

USA  Rsch  Ofc,  Durham,  ATTN  Life  Sciences  Dir 

USARIEM,  Natick.  ATTN  SGRD  UE  CA 

USATTC.  Ft  Clayton.  ATTN  STETC  MO  A 

USAIMA.  Ft  Bragg.  ATTN  ATSU  CTD  OM 

USAIMA.  Ft  Bragq,  ATTN  Marquat  Lib 

US  WAC  Ctr  & Sch.  Ft  McClellan.  ATTN:  Lib 

US  WAC  Ctr  & Sch.  Ft  McClellan.  ATTN:  Tng  Dir 

USA  Qujrtermaster  Sch.  Ft  Lee.  ATTN:  ATSM  TE 

Intelligence  Material  Dev  Ofc.  EWL.  Ft  Holabird 

USA  SE  Signal  Sch,  Ft  Gordon.  ATTN  ATSO  EA 

USA  Chaplain  Ctr  & Sch,  Ft  Hamilton.  ATTN:  ATSC-TERD 

USATSCH,  Ft  Eustis,  ATTN  Educ  Advisor 

USA  War  College.  Carlisle  Barracks,  ATTN:  Lib 

WRAIR.  Neuropsychiatry  Div 

DLI.  SDA,  Monterey 

USA  Concept  Anal  Agcy,  Bethesda.  ATTN:  MOCA  WGC 

USA  Concept  Anal  Agcy.  Bethesda,  ATTN-.  MOCA-MR 

USA  Concept  Anal  Agcy,  Bethesda.  ATTN:  MOCAJF 

USA  Artie  Test  Ctr.  APO  Seattle,  ATTN:  STEAC  MO  ASL 

USA  Attic  Test  Ctr.  APO  Seattle.  ATTN:  AMSTE  PL-TS 

USA  Armament  Cmd,  Redstone  Arsenal.  ATTN:  ATSKTEM 

USA  Armament  Cmd,  Rock  Island.  ATTN:  AMSAR-TDC 

FAA  NAFEC.  Atlantic  City,  ATTN-  Library 

FAA  NAFEC,  Atlantic  City.  ATTN:  Hum  Engr  Br 

FAA  Aeronautical  Ctr,  Oklahoma  City,  ATTN:  AAC  44D 

USA  Fid  Arty  Sch,  Ft  Sill,  ATTN:  Library 

USA  Armor  Sch.  Ft  Knox,  ATTN.  Library 

USA  Armor  Sch.  Ft  Knox.  ATTN  ATSB  DI  E 

USA  Armor  Sch.  Ft  Knox.  ATTN:  ATSB  DT  TP 

USA  Armor  Sch.  Ft  Knox,  ATTN:  ATSB  CD  AD 


2 HQUSACDEC,  Ft  Ord,  ATTN:  Library 

1 HQUSACDEC.  Ft  Ord.  ATTN:  ATEC-EX-E-Hum  Factors 

2 USAEEC,  Ft  Benjamin  Harrison,  ATTN:  Library 

1 USAPACDC,  Ft  Benjamin  Harrison,  ATTN:  ATCP-HR 
1 USA  Comm-Elect  Sch,  Ft  Monmouth,  ATTN:  ATSN-EA 
1 USAEC,  Ft  Monmouth.  ATTN.  AMSEL-CT  -HOP 
1 USAEC.  Ft  Monmouth.  ATTN:  AMSEL-PA-P 
1 USAEC.  Ft  Monmouth.  ATTN:  AMSEL  -SI  CB 
1 USAEC.  Ft  Monmouth,  ATTN  C.  Facl  Dev  Br 
1 USA  Materials  Sys  Anal  Agcy,  Aberdeen,  ATTN:  AMXSY-P 
1 Edgewood  Arsenal,  Aberdeen.  ATTN:  SAREA-BL-H 

1 USA  Ord  Ctr  & Sch,  Aberdeen.  ATTN:  ATSL-TEM— C 

2 USA  Hum  Engr  Lab.  Aberdeen.  ATTN:  Library/Dir 

1 USA  Combat  Arms  Tng  Bd,  Ft  Benning,  ATTN:  Ad  Supervisor 
1 USA  Infantry  Hum  Rsch  Unit,  Ft  Benning,  ATTN:  Chief 
1 USA  Infantry  Bd.  Ft  Benning.  ATTN  STEBC-TE  -T 
1 USASMA.  Ft  Bliss.  ATTN:  ATSS  LRC 
1 USA  Air  Def  Sch,  Ft  Bliss.  ATTN  ATSA  CTD  ME 
1 USA  Air  Def  Sch.  Ft  Bliss.  ATTN:  Tech  Lib 
1 USA  Air  Def  Bd.  Ft  Bliss.  ATTN:  FILES 
1 USA  Air  Def  Bd,  Ft  Bliss.  ATTN  STEBO-  PO 
1 USA  Cmd  & General  Stf  College.  Ft  Leavenworth,  ATTN:  Lib 
1 USA  Cmd  & General  Stf  College,  Ft  Leavenworth,  ATTN:  ATSW-SE-L 
1 USA  Cmd  & General  Stf  College,  Ft  Leavenworth,  ATTN:  Ed  Advisor 
1 USA  Combined  Arms  Cmbt  Dev  Act,  Ft  Leavenworth,  ATTN:  DepCdr 
1 USA  Combined  Arms  Cmbt  Dev  Act.  Ft  Leavenworth,  ATTN:  CCS 
1 USA  Combined  Arms  Cmbt  Dev  Act,  Ft  Leavenworth,  ATTN:  ATCASA 
1 USA  Combined  Arms  Cmbt  Dev  Act.  Ft  Leavenworth.  ATTN:  ATCACO-  E 
1 USA  Combined  Arms  Cmbt  Dev  Act.  Ft  Leavenworth,  ATTN:  ATCACC  Cl 
1 USAECOM.  Night  Vision  Lab.  Ft  Belvo.r,  ATTN:  AMSEL -NV-SD 

3 USA  Computer  Sys  Cmd,  Ft  Belvoir,  ATTN:  Tech  Library 
1 USAMERDC,  Ft  Belvo.r.  ATTN  STSFB  DO 

1 USA  Eng  Sch.  Ft  Belvoir.  ATTN:  Library 
1 USA  Topographic  Lab,  Ft  Belvoir,  ATTN:  ETL  TD-S 
1 USA  Topographic  Lab.  Ft  Belvoir,  ATTN.  STINFO  Center 
1 USA  Topographic  Lab,  Ft  Belvoir.  ATTN:  ETL-GSL 
1 USA  Intelligence  Ctr  & Sch.  Ft  Huachuca,  ATTN:  CTD-MS 

1 USA  Intelligence  Ctr  & Sch.  Ft  Huachuca.  ATTN:  ATS-CTD— MS 

1 USA  Intelligence  Ctr  & Sch.  Ft  Huachuca.  ATTN:  ATSI-TE 

1 USA  Intelligence  Ctr  & Sch,  Ft  Huachuca.  ATTN.  ATSI-TEX-GS 

1 USA  Intelligence  Ctr  & Sch.  Ft  Huachuca.  ATTN:  ATSI-CTS-OR 

1 USA  Intelligence  Ctr  & Sch,  Ft  Huachuca,  ATTN:  ATSI-CTD-DT 

1 USA  Intelligence  Ctr  & Sch,  Ft  Huachuca,  ATTN:  ATSI-CTD— CS 

1 USA  Intelligence  Ctr  & Sch,  Ft  Huachuca,  ATTN:  DAS/SRD 

1 USA  Intelligence  Ctr  & Sch,  Ft  Huachuca,  ATTN:  ATSI-TEM 

1 USA  Intelligence  Ctr  & Sch.  Ft  Huachuca.  ATTN:  Library 

1 CDR,  HQ  Ft  Huachuca.  ATTN  Tech  Ref  Div 

2 CDR.  USA  Electronic  Prvg  Grd.  ATTN:  STEEP-MT-S 
1 CDR.  Project  MASSTER,  ATTN:  Tech  Info  Center 

1 Hq  MASSTER.  USATRADOC.  LNO 
1 Research  Institute,  HQ  MASSTER.  Ft  Hood 
1 USA  Recruiting  Cmd,  Ft  Sherdtan,  ATTN:  USARCPM-P 
1 Senior  Army  Adv.,  USAFAGOD/TAC.  Elgin  AF  Aux  Fid  No  9 
1 HQ  USARPAC.  DCSPER,  APO  SF  96558,  ATTN  GPPE  -SE 
1 Stimson  Lib,  Academy  of  Health  Sciences,  Ft  Sam  Houston 
1 Marine  Corps  Inst.,  ATTN:  Dean-MCI 
1 HQUSMC.  Commandant.  ATTN:  Code  MTMT  51 

1 HQUSMC.  Commandant  ATTN  . Corfe  MPl-20 

2 USCG  Academy,  New  London.  ATTN:  Admission 
2 USCG  Academy,  New  London,  ATTN:  Library 

1 USCG  Training  Ctr.  NY,  ATTN:  CO 
1 USCG  Training  Ctr.  NY.  ATTN:  Educ  Svc  Ofc 
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